2024-03-23T15:29:40.535 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T15:29:40.535 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460 --name pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --description fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} -- /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/orig.config.yaml 2024-03-23T15:29:40.582 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T15:29:40.685 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460 branch: wip-batrick-testing-20240323.003144-squid description: fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} email: pdonnell@redhat.com first_in_suite: false job_id: '7618460' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request - SLOW_OPS - MON_DOWN sha1: 21b3bd672163092b4e542f772797fcba02d3d4de valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes check-counter: counters: mds: - mds.exported - mds.imported install: ceph: debuginfo: true flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARsO9mkf9Sa31l8h16xrU7JaFkB1i1hbbhND5Jh393VvdrfJQgScMrL5q2WDTpBNPcCZECoHnbd2bnmqxUjiXQ= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+9px7MnSxpoSkDwUcXr7nRW8kSItohx68roYhA2tWLsD7fcV7x+aE719oUsVNHglGXaCzm4ON+32TRNsGUUKg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-23_09:56:53 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T15:29:40.685 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-23T15:29:40.686 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-23T15:29:40.686 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T15:29:40.688 INFO:teuthology.task.internal:Checking packages... 2024-03-23T15:29:40.712 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-23T15:29:40.712 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T15:29:40.712 INFO:teuthology.packaging:ref: None 2024-03-23T15:29:40.712 INFO:teuthology.packaging:tag: None 2024-03-23T15:29:40.712 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-23T15:29:40.712 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T15:29:40.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-23T15:29:40.854 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-23T15:29:40.855 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T15:29:40.862 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T15:29:40.862 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T15:29:40.885 INFO:teuthology.task.internal:Saving configuration 2024-03-23T15:29:40.902 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T15:29:40.908 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T15:29:40.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 15:23:59.582309', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARsO9mkf9Sa31l8h16xrU7JaFkB1i1hbbhND5Jh393VvdrfJQgScMrL5q2WDTpBNPcCZECoHnbd2bnmqxUjiXQ='} 2024-03-23T15:29:40.949 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 15:23:59.580885', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+9px7MnSxpoSkDwUcXr7nRW8kSItohx68roYhA2tWLsD7fcV7x+aE719oUsVNHglGXaCzm4ON+32TRNsGUUKg='} 2024-03-23T15:29:40.949 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T15:29:40.955 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-23T15:29:40.955 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-23T15:29:40.955 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T15:29:41.073 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a6fea2160>, signals=[15]) 2024-03-23T15:29:41.073 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T15:29:41.080 INFO:teuthology.task.internal:Opening connections... 2024-03-23T15:29:41.080 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T15:29:41.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:29:41.174 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T15:29:41.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:29:41.262 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T15:29:41.272 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-23T15:29:41.289 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-23T15:29:41.290 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-23T15:29:41.346 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-03-23T15:29:41.346 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-03-23T15:29:41.346 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-03-23T15:29:41.346 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-03-23T15:29:41.346 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-03-23T15:29:41.346 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T15:29:41.347 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T15:29:41.348 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-23T15:29:41.373 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-03-23T15:29:41.390 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-03-23T15:29:41.391 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T15:29:41.448 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-03-23T15:29:41.449 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T15:29:41.449 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T15:29:41.449 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T15:29:41.449 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-03-23T15:29:41.473 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T15:29:41.484 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T15:29:41.497 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T15:29:41.497 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-23T15:29:41.499 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-03-23T15:29:41.515 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T15:29:41.534 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T15:29:41.534 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-23T15:29:41.556 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-03-23T15:29:41.746 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T15:29:41.806 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T15:29:41.825 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-23T15:29:41.825 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-23T15:29:41.825 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T15:29:41.826 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T15:29:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T15:29:41.842 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-23T15:29:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T15:29:41.843 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-23T15:29:41.899 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-361.el9.x86_64 2024-03-23T15:29:41.899 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-361.el9.x86_64 2024-03-23T15:29:41.899 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-03-23T15:29:41.900 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-03-23T15:29:41.900 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-03-23T15:29:41.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-03-23T15:29:44.017 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-23T15:29:44.033 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-23T15:29:45.401 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-23T15:29:46.149 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-03-23T15:29:51.742 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 311 kB/s | 2.0 MB 00:06 2024-03-23T15:29:52.814 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-23T15:29:57.040 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-03-23T15:30:01.577 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-23T15:30:03.000 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-03-23T15:30:04.026 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-23T15:30:07.890 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 932 kB/s | 19 MB 00:20 2024-03-23T15:30:09.018 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-23T15:30:09.186 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:01 ago on Sat 23 Mar 2024 03:30:08 PM UTC. 2024-03-23T15:30:11.149 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-23T15:30:11.149 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-23T15:30:11.235 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-23T15:30:11.241 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-23T15:30:11.241 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-23T15:30:11.241 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-03-23T15:30:11.242 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:30:11.243 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 76 M 2024-03-23T15:30:11.243 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 123 M 2024-03-23T15:30:11.243 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-23T15:30:11.344 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 754 kB/s | 44 kB 00:00 2024-03-23T15:30:11.928 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-23T15:30:12.132 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-23T15:30:12.586 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-23T15:30:12.870 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-23T15:30:12.870 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-23T15:30:12.870 INFO:teuthology.orchestra.run.smithi157.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-23T15:30:13.539 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-23T15:30:13.691 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-23T15:30:13.909 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-23T15:30:13.909 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-23T15:30:14.485 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-23T15:30:14.914 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-23T15:30:14.914 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-23T15:30:17.400 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-23T15:30:18.077 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-23T15:30:18.178 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T15:30:19.368 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-23T15:30:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T15:30:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T15:30:21.441 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-23T15:30:21.441 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-23T15:30:21.539 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-23T15:30:21.548 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-23T15:30:21.548 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-23T15:30:21.548 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 76 M 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-03-23T15:30:21.549 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-23T15:30:21.786 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 377 kB/s | 44 kB 00:00 2024-03-23T15:30:22.353 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-23T15:30:22.903 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-23T15:30:23.353 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-23T15:30:23.353 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-23T15:30:23.354 INFO:teuthology.orchestra.run.smithi129.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-23T15:30:24.025 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-23T15:30:24.406 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-23T15:30:24.406 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-23T15:30:25.412 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-23T15:30:25.412 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-23T15:30:25.756 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T15:30:25.799 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:30:27.881 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-23T15:30:28.479 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-23T15:30:28.562 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T15:30:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:30:31.320 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T15:30:31.487 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T15:30:36.234 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T15:30:36.345 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:30:41.103 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:30:59.812 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:30:59.813 INFO:teuthology.orchestra.run.smithi157.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T15:30:59.813 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:30:59.858 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:00.239 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:00.240 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-23T15:31:00.240 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T15:31:00.240 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-23T15:31:00.632 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:31:00.633 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-23T15:31:00.952 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T15:31:01.051 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T15:31:01.051 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T15:31:01.051 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-23T15:31:01.051 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-23T15:31:01.051 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-23T15:31:01.052 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-23T15:31:01.052 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-03-23T15:31:01.052 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T15:31:01.132 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T15:31:01.132 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T15:31:01.132 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 68.el9 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T15:31:01.166 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8264254 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T15:31:01.167 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-03-23T15:31:01.168 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:31:01.168 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T15:31:01.168 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T15:31:01.168 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-430.el9.x86_64 2024-03-23T15:31:01.169 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T15:31:01.788 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-03-23T15:31:03.119 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T15:31:03.152 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-03-23T15:31:03.155 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-23T15:31:03.170 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.mF8YrZkHlL 2024-03-23T15:31:03.170 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mF8YrZkHlL 2024-03-23T15:31:03.240 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.mF8YrZkHlL 2024-03-23T15:31:03.381 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.mF8YrZkHlL is 6KB 2024-03-23T15:31:03.429 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.mF8YrZkHlL 2024-03-23T15:31:03.443 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T15:31:03.510 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T15:31:03.511 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T15:31:03.511 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T15:31:03.511 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-23T15:31:03.512 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-23T15:31:03.663 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-03-23T15:31:10.392 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:10.393 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T15:31:10.393 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:31:10.438 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:10.781 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:10.781 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-23T15:31:10.781 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T15:31:10.781 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:31:11.206 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-23T15:31:11.547 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T15:31:11.635 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T15:31:11.635 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T15:31:11.635 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-23T15:31:11.635 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-23T15:31:11.636 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-23T15:31:11.636 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-23T15:31:11.636 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-23T15:31:11.636 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T15:31:11.715 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T15:31:11.715 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T15:31:11.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T15:31:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:31:11.757 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T15:31:11.758 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T15:31:11.758 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-430.el9.x86_64 2024-03-23T15:31:11.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T15:31:12.348 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-23T15:31:13.850 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T15:31:13.876 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-23T15:31:13.878 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-23T15:31:13.893 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.90yGi1ZEFO 2024-03-23T15:31:13.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.90yGi1ZEFO 2024-03-23T15:31:13.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.90yGi1ZEFO 2024-03-23T15:31:14.108 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.90yGi1ZEFO is 6KB 2024-03-23T15:31:14.160 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.90yGi1ZEFO 2024-03-23T15:31:14.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T15:31:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T15:31:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T15:31:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T15:31:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-23T15:31:14.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-23T15:31:14.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-23T15:31:33.685 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-23T15:31:33.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:31:33.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:31:44.414 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-23T15:31:44.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-23T15:31:44.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:32:08.351 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-23T15:32:17.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:32:17.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:32:19.103 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-23T15:32:20.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-23T15:32:28.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-23T15:32:28.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:32:32.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:32:32.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:32:35.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-23T15:32:46.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-23T15:32:50.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:32:50.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:32:53.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-23T15:32:58.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-23T15:32:58.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:32:58.904 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-23T15:32:59.408 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-23T15:32:59.408 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-23T15:32:59.408 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-23T15:32:59.422 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-430.el9.x86_64 2024-03-23T15:32:59.422 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-23T15:32:59.422 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T15:32:59.422 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T15:33:00.423 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-23T15:33:00.424 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T15:33:00.485 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-23T15:33:00.527 DEBUG:teuthology.parallel:result is None 2024-03-23T15:33:11.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:33:11.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:33:11.922 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-23T15:33:12.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:33:12.290 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-23T15:33:12.290 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-23T15:33:12.305 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-430.el9.x86_64 2024-03-23T15:33:12.305 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-23T15:33:12.306 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T15:33:12.306 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T15:33:13.307 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-23T15:33:13.307 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T15:33:13.382 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-03-23T15:33:13.410 DEBUG:teuthology.parallel:result is None 2024-03-23T15:33:13.411 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T15:33:13.418 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T15:33:13.418 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T15:33:13.421 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T15:33:13.439 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T15:33:13.446 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T15:33:13.457 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T15:33:13.458 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T15:33:13.481 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T15:33:13.576 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T15:33:13.583 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T15:33:13.583 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T15:33:13.585 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T15:33:13.616 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T15:33:13.620 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T15:33:13.631 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T15:33:13.634 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T15:33:13.636 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T15:33:13.644 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T15:33:13.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T15:33:13.676 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T15:33:13.706 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T15:33:13.716 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T15:33:13.716 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T15:33:13.747 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T15:33:13.763 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T15:33:13.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T15:33:13.903 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T15:33:13.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T15:33:13.988 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:33:13.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T15:33:14.056 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T15:33:14.097 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T15:33:14.158 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T15:33:14.244 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T15:33:14.288 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:33:14.288 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T15:33:14.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-23T15:33:14.360 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-03-23T15:33:14.419 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T15:33:14.454 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T15:33:14.959 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T15:33:14.968 INFO:teuthology.task.internal:Starting timer... 2024-03-23T15:33:14.968 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T15:33:14.979 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T15:33:15.005 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-23T15:33:15.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-03-23T15:33:15.073 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-03-23T15:33:15.076 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-03-23T15:33:15.141 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-03-23T15:33:15.143 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T15:33:15.143 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-03-23T15:33:15.168 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-03-23T15:33:15.168 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-03-23T15:33:15.193 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-03-23T15:33:15.193 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-03-23T15:33:15.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T15:33:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T15:33:15.237 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T15:33:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T15:33:15.267 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T15:33:15.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-03-23T15:33:15.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-03-23T15:33:15.347 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T15:33:15.357 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T15:33:15.373 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T15:33:15.374 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T15:37:27.919 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-03-23T15:37:27.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-23T15:37:27.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:37:28.013 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-23T15:37:28.088 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-23T15:37:28.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:37:28.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T15:37:28.177 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-23T15:37:28.253 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-23T15:37:28.253 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T15:37:28.264 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T15:37:28.264 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T15:37:28.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T15:37:28.267 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T15:37:28.267 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T15:37:28.296 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T15:37:28.312 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T15:37:28.323 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T15:37:28.340 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T15:37:28.341 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-03-23T15:37:28.355 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-03-23T15:37:28.369 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-03-23T15:37:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T15:37:28.383 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-03-23T15:37:28.387 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T15:37:28.400 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T15:37:28.416 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T15:37:28.444 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-23T15:37:28.447 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T15:37:28.447 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-23T15:37:28.447 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.447 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.447 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.447 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.479 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-23T15:37:28.483 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T15:37:28.483 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-23T15:37:28.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T15:37:28.483 INFO:teuthology.run_tasks:Running task install... 2024-03-23T15:37:28.492 DEBUG:teuthology.task.install:project ceph 2024-03-23T15:37:28.492 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-23T15:37:28.492 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-23T15:37:28.492 INFO:teuthology.task.install:Using flavor: default 2024-03-23T15:37:28.498 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-23T15:37:28.498 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-23T15:37:28.499 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-23T15:37:28.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T15:37:28.501 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-23T15:37:28.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T15:37:28.647 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-23T15:37:28.647 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-23T15:37:28.719 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-23T15:37:28.719 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-23T15:37:28.822 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T15:37:28.822 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:37:28.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T15:37:28.851 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T15:37:28.851 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T15:37:28.899 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T15:37:28.899 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:37:28.899 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T15:37:28.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T15:37:28.928 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T15:37:28.928 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T15:37:28.997 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T15:37:28.998 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T15:37:29.065 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-03-23T15:37:29.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-03-23T15:37:29.077 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T15:37:29.143 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-03-23T15:37:29.145 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-03-23T15:37:29.310 INFO:teuthology.orchestra.run.smithi129.stdout:74 files removed 2024-03-23T15:37:29.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T15:37:29.395 INFO:teuthology.orchestra.run.smithi157.stdout:74 files removed 2024-03-23T15:37:29.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T15:37:30.042 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-03-23T15:37:30.189 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-03-23T15:37:30.484 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-23T15:37:30.573 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-23T15:37:30.860 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-23T15:37:30.890 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-23T15:37:31.301 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-23T15:37:31.612 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-23T15:37:32.693 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-23T15:37:33.112 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-23T15:37:33.742 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-23T15:37:35.011 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-23T15:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-23T15:37:39.888 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-23T15:37:42.353 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-23T15:37:44.630 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-23T15:37:49.662 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-23T15:37:51.319 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-23T15:37:52.015 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T15:37:52.015 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T15:37:52.048 INFO:teuthology.orchestra.run.smithi157.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T15:37:52.048 INFO:teuthology.orchestra.run.smithi157.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T15:37:52.049 INFO:teuthology.orchestra.run.smithi157.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T15:37:52.050 INFO:teuthology.orchestra.run.smithi157.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T15:37:52.052 INFO:teuthology.orchestra.run.smithi157.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T15:37:52.054 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T15:37:52.055 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T15:37:52.139 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout:==================================================================================================== 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout:==================================================================================================== 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 68 M 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 437 M 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-23T15:37:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.0 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 53 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 20 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 287 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 176 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-23T15:37:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 674 M 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.4 M 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 109 k 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 44 M 2024-03-23T15:37:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 52 M 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 615 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 770 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.1 M 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 337 k 2024-03-23T15:37:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 509 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 51 M 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.3 M 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T15:37:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T15:37:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 22 M 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-23T15:37:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-23T15:37:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T15:37:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T15:37:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T15:37:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T15:37:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T15:37:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout:==================================================================================================== 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout:Install 172 Packages 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 24 Packages 2024-03-23T15:37:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:37:52.174 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.2 G 2024-03-23T15:37:52.174 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-23T15:37:52.888 INFO:teuthology.orchestra.run.smithi157.stdout:(1/196): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-23T15:37:53.455 INFO:teuthology.orchestra.run.smithi157.stdout:(2/196): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 7.3 MB/s | 5.3 MB 00:00 2024-03-23T15:37:56.695 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-23T15:37:57.374 INFO:teuthology.orchestra.run.smithi157.stdout:(3/196): ceph-common-19.0.0-1855.g21b3bd67.el9. 4.6 MB/s | 21 MB 00:04 2024-03-23T15:37:58.246 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-23T15:37:59.372 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-23T15:37:59.813 INFO:teuthology.orchestra.run.smithi157.stdout:(4/196): ceph-base-debuginfo-19.0.0-1855.g21b3b 9.6 MB/s | 68 MB 00:07 2024-03-23T15:37:59.980 INFO:teuthology.orchestra.run.smithi157.stdout:(5/196): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 5.2 MB/s | 888 kB 00:00 2024-03-23T15:38:00.739 INFO:teuthology.orchestra.run.smithi157.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-1855.g21b3b 7.9 MB/s | 6.0 MB 00:00 2024-03-23T15:38:00.814 INFO:teuthology.orchestra.run.smithi157.stdout:(7/196): ceph-immutable-object-cache-19.0.0-185 1.9 MB/s | 145 kB 00:00 2024-03-23T15:38:01.048 INFO:teuthology.orchestra.run.smithi157.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 6.2 MB/s | 1.5 MB 00:00 2024-03-23T15:38:01.398 INFO:teuthology.orchestra.run.smithi157.stdout:(9/196): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 6.7 MB/s | 2.4 MB 00:00 2024-03-23T15:38:03.273 INFO:teuthology.orchestra.run.smithi157.stdout:(10/196): ceph-debuginfo-19.0.0-1855.g21b3bd67. 3.8 MB/s | 22 MB 00:05 2024-03-23T15:38:04.165 INFO:teuthology.orchestra.run.smithi157.stdout:(11/196): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x8 1.7 MB/s | 1.5 MB 00:00 2024-03-23T15:38:04.247 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-23T15:38:06.631 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T15:38:06.632 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T15:38:06.668 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T15:38:06.669 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T15:38:06.670 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T15:38:06.671 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T15:38:06.674 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T15:38:06.677 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T15:38:06.678 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T15:38:06.762 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-23T15:38:06.780 INFO:teuthology.orchestra.run.smithi129.stdout:==================================================================================================== 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout:==================================================================================================== 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 68 M 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 437 M 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.0 M 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-23T15:38:06.781 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 53 M 2024-03-23T15:38:06.782 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-23T15:38:06.782 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-23T15:38:06.782 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-23T15:38:06.782 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 20 M 2024-03-23T15:38:06.785 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 287 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 176 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 674 M 2024-03-23T15:38:06.786 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.4 M 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 109 k 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 44 M 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 52 M 2024-03-23T15:38:06.787 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 615 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 770 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.1 M 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 337 k 2024-03-23T15:38:06.788 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 509 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 51 M 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.3 M 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T15:38:06.789 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T15:38:06.790 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 22 M 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-23T15:38:06.791 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T15:38:06.792 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T15:38:06.793 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T15:38:06.794 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T15:38:06.795 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T15:38:06.796 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T15:38:06.797 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T15:38:06.798 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout:==================================================================================================== 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout:Install 172 Packages 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 24 Packages 2024-03-23T15:38:06.799 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:38:06.800 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.2 G 2024-03-23T15:38:06.800 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-23T15:38:07.534 INFO:teuthology.orchestra.run.smithi129.stdout:(1/196): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-23T15:38:08.917 INFO:teuthology.orchestra.run.smithi129.stdout:(2/196): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 3.0 MB/s | 5.3 MB 00:01 2024-03-23T15:38:12.023 INFO:teuthology.orchestra.run.smithi157.stdout:(12/196): ceph-mds-debuginfo-19.0.0-1855.g21b3b 4.9 MB/s | 53 MB 00:10 2024-03-23T15:38:13.407 INFO:teuthology.orchestra.run.smithi157.stdout:(13/196): ceph-mon-19.0.0-1855.g21b3bd67.el9.x8 3.4 MB/s | 4.7 MB 00:01 2024-03-23T15:38:19.573 INFO:teuthology.orchestra.run.smithi157.stdout:(14/196): ceph-mgr-debuginfo-19.0.0-1855.g21b3b 1.3 MB/s | 20 MB 00:15 2024-03-23T15:38:35.346 INFO:teuthology.orchestra.run.smithi157.stdout:(15/196): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 1.1 MB/s | 17 MB 00:15 2024-03-23T15:38:37.340 INFO:teuthology.orchestra.run.smithi129.stdout:(3/196): ceph-common-19.0.0-1855.g21b3bd67.el9. 706 kB/s | 21 MB 00:29 2024-03-23T15:38:48.064 INFO:teuthology.orchestra.run.smithi129.stdout:(4/196): ceph-base-debuginfo-19.0.0-1855.g21b3b 1.7 MB/s | 68 MB 00:40 2024-03-23T15:38:48.606 INFO:teuthology.orchestra.run.smithi129.stdout:(5/196): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.6 MB/s | 888 kB 00:00 2024-03-23T15:38:51.415 INFO:teuthology.orchestra.run.smithi129.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-1855.g21b3b 2.1 MB/s | 6.0 MB 00:02 2024-03-23T15:38:51.558 INFO:teuthology.orchestra.run.smithi129.stdout:(7/196): ceph-immutable-object-cache-19.0.0-185 1.0 MB/s | 145 kB 00:00 2024-03-23T15:38:52.215 INFO:teuthology.orchestra.run.smithi129.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 2.2 MB/s | 1.5 MB 00:00 2024-03-23T15:38:53.257 INFO:teuthology.orchestra.run.smithi129.stdout:(9/196): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 2.3 MB/s | 2.4 MB 00:01 2024-03-23T15:38:55.040 INFO:teuthology.orchestra.run.smithi129.stdout:(10/196): ceph-debuginfo-19.0.0-1855.g21b3bd67. 1.3 MB/s | 22 MB 00:17 2024-03-23T15:38:55.194 INFO:teuthology.orchestra.run.smithi157.stdout:(16/196): ceph-mon-debuginfo-19.0.0-1855.g21b3b 2.4 MB/s | 99 MB 00:41 2024-03-23T15:38:55.723 INFO:teuthology.orchestra.run.smithi129.stdout:(11/196): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x8 2.2 MB/s | 1.5 MB 00:00 2024-03-23T15:38:58.627 INFO:teuthology.orchestra.run.smithi157.stdout:(17/196): ceph-radosgw-19.0.0-1855.g21b3bd67.el 2.8 MB/s | 9.8 MB 00:03 2024-03-23T15:39:02.765 INFO:teuthology.orchestra.run.smithi129.stdout:(12/196): ceph-mgr-debuginfo-19.0.0-1855.g21b3b 2.8 MB/s | 20 MB 00:07 2024-03-23T15:39:04.281 INFO:teuthology.orchestra.run.smithi129.stdout:(13/196): ceph-mon-19.0.0-1855.g21b3bd67.el9.x8 3.1 MB/s | 4.7 MB 00:01 2024-03-23T15:39:11.156 INFO:teuthology.orchestra.run.smithi129.stdout:(14/196): ceph-mds-debuginfo-19.0.0-1855.g21b3b 2.9 MB/s | 53 MB 00:17 2024-03-23T15:39:16.597 INFO:teuthology.orchestra.run.smithi129.stdout:(15/196): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 3.1 MB/s | 17 MB 00:05 2024-03-23T15:39:27.573 INFO:teuthology.orchestra.run.smithi157.stdout:(18/196): ceph-radosgw-debuginfo-19.0.0-1855.g2 6.1 MB/s | 176 MB 00:28 2024-03-23T15:39:27.673 INFO:teuthology.orchestra.run.smithi157.stdout:(19/196): ceph-selinux-19.0.0-1855.g21b3bd67.el 252 kB/s | 25 kB 00:00 2024-03-23T15:39:33.571 INFO:teuthology.orchestra.run.smithi129.stdout:(16/196): ceph-mon-debuginfo-19.0.0-1855.g21b3b 3.4 MB/s | 99 MB 00:29 2024-03-23T15:39:34.914 INFO:teuthology.orchestra.run.smithi157.stdout:(20/196): ceph-test-19.0.0-1855.g21b3bd67.el9.x 6.5 MB/s | 47 MB 00:07 2024-03-23T15:39:37.381 INFO:teuthology.orchestra.run.smithi157.stdout:(21/196): ceph-common-debuginfo-19.0.0-1855.g21 4.2 MB/s | 437 MB 01:43 2024-03-23T15:39:37.531 INFO:teuthology.orchestra.run.smithi157.stdout:(22/196): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.4 MB/s | 219 kB 00:00 2024-03-23T15:39:37.570 INFO:teuthology.orchestra.run.smithi129.stdout:(17/196): ceph-radosgw-19.0.0-1855.g21b3bd67.el 2.4 MB/s | 9.8 MB 00:03 2024-03-23T15:39:37.598 INFO:teuthology.orchestra.run.smithi157.stdout:(23/196): cephfs-shell-19.0.0-1855.g21b3bd67.el 407 kB/s | 27 kB 00:00 2024-03-23T15:39:37.724 INFO:teuthology.orchestra.run.smithi157.stdout:(24/196): libcephfs-devel-19.0.0-1855.g21b3bd67 254 kB/s | 32 kB 00:00 2024-03-23T15:39:38.066 INFO:teuthology.orchestra.run.smithi157.stdout:(25/196): libcephfs2-19.0.0-1855.g21b3bd67.el9. 2.1 MB/s | 744 kB 00:00 2024-03-23T15:39:39.274 INFO:teuthology.orchestra.run.smithi157.stdout:(26/196): libcephfs2-debuginfo-19.0.0-1855.g21b 4.4 MB/s | 5.4 MB 00:01 2024-03-23T15:39:39.374 INFO:teuthology.orchestra.run.smithi157.stdout:(27/196): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.6 MB/s | 163 kB 00:00 2024-03-23T15:39:39.466 INFO:teuthology.orchestra.run.smithi157.stdout:(28/196): librados-devel-19.0.0-1855.g21b3bd67. 1.4 MB/s | 127 kB 00:00 2024-03-23T15:39:39.567 INFO:teuthology.orchestra.run.smithi157.stdout:(29/196): librados-devel-debuginfo-19.0.0-1855. 1.1 MB/s | 109 kB 00:00 2024-03-23T15:39:50.740 INFO:teuthology.orchestra.run.smithi157.stdout:(30/196): librados2-debuginfo-19.0.0-1855.g21b3 4.0 MB/s | 44 MB 00:11 2024-03-23T15:39:50.857 INFO:teuthology.orchestra.run.smithi157.stdout:(31/196): libradosstriper1-19.0.0-1855.g21b3bd6 4.2 MB/s | 499 kB 00:00 2024-03-23T15:39:51.820 INFO:teuthology.orchestra.run.smithi129.stdout:(18/196): ceph-common-debuginfo-19.0.0-1855.g21 4.2 MB/s | 437 MB 01:42 2024-03-23T15:39:52.220 INFO:teuthology.orchestra.run.smithi129.stdout:(19/196): ceph-selinux-19.0.0-1855.g21b3bd67.el 63 kB/s | 25 kB 00:00 2024-03-23T15:39:59.881 INFO:teuthology.orchestra.run.smithi157.stdout:(32/196): ceph-osd-debuginfo-19.0.0-1855.g21b3b 3.4 MB/s | 287 MB 01:24 2024-03-23T15:40:03.889 INFO:teuthology.orchestra.run.smithi157.stdout:(33/196): librgw2-19.0.0-1855.g21b3bd67.el9.x86 1.3 MB/s | 5.1 MB 00:04 2024-03-23T15:40:03.964 INFO:teuthology.orchestra.run.smithi157.stdout:(34/196): python3-ceph-argparse-19.0.0-1855.g21 603 kB/s | 45 kB 00:00 2024-03-23T15:40:04.115 INFO:teuthology.orchestra.run.smithi157.stdout:(35/196): python3-ceph-common-19.0.0-1855.g21b3 875 kB/s | 131 kB 00:00 2024-03-23T15:40:04.273 INFO:teuthology.orchestra.run.smithi157.stdout:(36/196): python3-cephfs-19.0.0-1855.g21b3bd67. 1.0 MB/s | 161 kB 00:00 2024-03-23T15:40:04.732 INFO:teuthology.orchestra.run.smithi157.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-1855. 1.3 MB/s | 615 kB 00:00 2024-03-23T15:40:04.982 INFO:teuthology.orchestra.run.smithi157.stdout:(38/196): python3-rados-19.0.0-1855.g21b3bd67.e 1.3 MB/s | 320 kB 00:00 2024-03-23T15:40:05.532 INFO:teuthology.orchestra.run.smithi157.stdout:(39/196): python3-rados-debuginfo-19.0.0-1855.g 1.4 MB/s | 770 kB 00:00 2024-03-23T15:40:05.741 INFO:teuthology.orchestra.run.smithi157.stdout:(40/196): python3-rbd-19.0.0-1855.g21b3bd67.el9 1.4 MB/s | 298 kB 00:00 2024-03-23T15:40:06.441 INFO:teuthology.orchestra.run.smithi157.stdout:(41/196): python3-rbd-debuginfo-19.0.0-1855.g21 1.6 MB/s | 1.1 MB 00:00 2024-03-23T15:40:06.525 INFO:teuthology.orchestra.run.smithi157.stdout:(42/196): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.2 MB/s | 99 kB 00:00 2024-03-23T15:40:06.775 INFO:teuthology.orchestra.run.smithi157.stdout:(43/196): python3-rgw-debuginfo-19.0.0-1855.g21 1.3 MB/s | 337 kB 00:00 2024-03-23T15:40:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:(44/196): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 854 kB/s | 85 kB 00:00 2024-03-23T15:40:07.167 INFO:teuthology.orchestra.run.smithi157.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-1855.g21b3b 1.7 MB/s | 509 kB 00:00 2024-03-23T15:40:07.461 INFO:teuthology.orchestra.run.smithi129.stdout:(20/196): ceph-osd-debuginfo-19.0.0-1855.g21b3b 5.7 MB/s | 287 MB 00:50 2024-03-23T15:40:08.801 INFO:teuthology.orchestra.run.smithi157.stdout:(46/196): librbd1-debuginfo-19.0.0-1855.g21b3bd 2.9 MB/s | 52 MB 00:17 2024-03-23T15:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout:(47/196): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 1.4 MB/s | 3.1 MB 00:02 2024-03-23T15:40:09.543 INFO:teuthology.orchestra.run.smithi157.stdout:(48/196): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 974 kB/s | 170 kB 00:00 2024-03-23T15:40:10.468 INFO:teuthology.orchestra.run.smithi157.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-1855.g21b3bd 1.4 MB/s | 1.3 MB 00:00 2024-03-23T15:40:10.551 INFO:teuthology.orchestra.run.smithi157.stdout:(50/196): ceph-grafana-dashboards-19.0.0-1855.g 379 kB/s | 31 kB 00:00 2024-03-23T15:40:10.710 INFO:teuthology.orchestra.run.smithi157.stdout:(51/196): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 872 kB/s | 138 kB 00:00 2024-03-23T15:40:11.652 INFO:teuthology.orchestra.run.smithi157.stdout:(52/196): ceph-mgr-dashboard-19.0.0-1855.g21b3b 1.9 MB/s | 1.8 MB 00:00 2024-03-23T15:40:15.835 INFO:teuthology.orchestra.run.smithi157.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 1.8 MB/s | 7.4 MB 00:04 2024-03-23T15:40:16.027 INFO:teuthology.orchestra.run.smithi157.stdout:(54/196): ceph-mgr-modules-core-19.0.0-1855.g21 1.3 MB/s | 248 kB 00:00 2024-03-23T15:40:16.102 INFO:teuthology.orchestra.run.smithi157.stdout:(55/196): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 659 kB/s | 49 kB 00:00 2024-03-23T15:40:16.177 INFO:teuthology.orchestra.run.smithi157.stdout:(56/196): ceph-prometheus-alerts-19.0.0-1855.g2 204 kB/s | 15 kB 00:00 2024-03-23T15:40:16.353 INFO:teuthology.orchestra.run.smithi157.stdout:(57/196): ceph-volume-19.0.0-1855.g21b3bd67.el9 1.5 MB/s | 264 kB 00:00 2024-03-23T15:40:16.703 INFO:teuthology.orchestra.run.smithi157.stdout:(58/196): cephadm-19.0.0-1855.g21b3bd67.el9.noa 2.1 MB/s | 748 kB 00:00 2024-03-23T15:40:16.770 INFO:teuthology.orchestra.run.smithi157.stdout:(59/196): cephfs-top-19.0.0-1855.g21b3bd67.el9. 274 kB/s | 18 kB 00:00 2024-03-23T15:40:17.020 INFO:teuthology.orchestra.run.smithi157.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-23T15:40:17.121 INFO:teuthology.orchestra.run.smithi157.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 9.0 MB/s | 920 kB 00:00 2024-03-23T15:40:17.188 INFO:teuthology.orchestra.run.smithi157.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-23T15:40:17.271 INFO:teuthology.orchestra.run.smithi157.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-23T15:40:17.338 INFO:teuthology.orchestra.run.smithi157.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 474 kB/s | 32 kB 00:00 2024-03-23T15:40:17.580 INFO:teuthology.orchestra.run.smithi157.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-03-23T15:40:17.622 INFO:teuthology.orchestra.run.smithi157.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 413 kB/s | 17 kB 00:00 2024-03-23T15:40:17.655 INFO:teuthology.orchestra.run.smithi157.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-23T15:40:17.706 INFO:teuthology.orchestra.run.smithi157.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-23T15:40:17.748 INFO:teuthology.orchestra.run.smithi157.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-23T15:40:17.807 INFO:teuthology.orchestra.run.smithi157.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-23T15:40:17.882 INFO:teuthology.orchestra.run.smithi157.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-23T15:40:17.924 INFO:teuthology.orchestra.run.smithi157.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-23T15:40:17.974 INFO:teuthology.orchestra.run.smithi157.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-23T15:40:18.016 INFO:teuthology.orchestra.run.smithi157.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-23T15:40:18.058 INFO:teuthology.orchestra.run.smithi157.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-23T15:40:18.092 INFO:teuthology.orchestra.run.smithi157.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-23T15:40:18.259 INFO:teuthology.orchestra.run.smithi157.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-23T15:40:18.334 INFO:teuthology.orchestra.run.smithi157.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-23T15:40:18.401 INFO:teuthology.orchestra.run.smithi157.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-23T15:40:18.443 INFO:teuthology.orchestra.run.smithi157.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-23T15:40:18.652 INFO:teuthology.orchestra.run.smithi157.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-03-23T15:40:18.727 INFO:teuthology.orchestra.run.smithi157.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-23T15:40:18.778 INFO:teuthology.orchestra.run.smithi157.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 78 kB 00:00 2024-03-23T15:40:18.986 INFO:teuthology.orchestra.run.smithi157.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-23T15:40:19.028 INFO:teuthology.orchestra.run.smithi157.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 586 kB/s | 24 kB 00:00 2024-03-23T15:40:19.078 INFO:teuthology.orchestra.run.smithi157.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 362 kB/s | 18 kB 00:00 2024-03-23T15:40:19.137 INFO:teuthology.orchestra.run.smithi157.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-23T15:40:19.187 INFO:teuthology.orchestra.run.smithi157.stdout:(88/196): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-23T15:40:19.412 INFO:teuthology.orchestra.run.smithi157.stdout:(89/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-23T15:40:19.463 INFO:teuthology.orchestra.run.smithi157.stdout:(90/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-23T15:40:19.521 INFO:teuthology.orchestra.run.smithi157.stdout:(91/196): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-23T15:40:19.580 INFO:teuthology.orchestra.run.smithi157.stdout:(92/196): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-23T15:40:19.630 INFO:teuthology.orchestra.run.smithi157.stdout:(93/196): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-23T15:40:19.689 INFO:teuthology.orchestra.run.smithi157.stdout:(94/196): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-23T15:40:19.739 INFO:teuthology.orchestra.run.smithi157.stdout:(95/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-23T15:40:19.798 INFO:teuthology.orchestra.run.smithi157.stdout:(96/196): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-23T15:40:19.848 INFO:teuthology.orchestra.run.smithi157.stdout:(97/196): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-23T15:40:19.899 INFO:teuthology.orchestra.run.smithi157.stdout:(98/196): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-23T15:40:20.732 INFO:teuthology.orchestra.run.smithi157.stdout:(99/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-03-23T15:40:20.782 INFO:teuthology.orchestra.run.smithi157.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-23T15:40:20.833 INFO:teuthology.orchestra.run.smithi157.stdout:(101/196): python3-wcwidth-0.2.5-8.el9.noarch.r 817 kB/s | 41 kB 00:00 2024-03-23T15:40:20.925 INFO:teuthology.orchestra.run.smithi157.stdout:(102/196): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-23T15:40:20.992 INFO:teuthology.orchestra.run.smithi157.stdout:(103/196): uuid-1.6.2-55.el9.x86_64.rpm 847 kB/s | 56 kB 00:00 2024-03-23T15:40:21.059 INFO:teuthology.orchestra.run.smithi157.stdout:(104/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 940 kB/s | 63 kB 00:00 2024-03-23T15:40:21.126 INFO:teuthology.orchestra.run.smithi157.stdout:(105/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 661 kB/s | 44 kB 00:00 2024-03-23T15:40:21.168 INFO:teuthology.orchestra.run.smithi157.stdout:(106/196): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-23T15:40:21.235 INFO:teuthology.orchestra.run.smithi157.stdout:(107/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-23T15:40:21.360 INFO:teuthology.orchestra.run.smithi157.stdout:(108/196): python3-cryptography-36.0.1-4.el9.x8 9.2 MB/s | 1.1 MB 00:00 2024-03-23T15:40:21.427 INFO:teuthology.orchestra.run.smithi157.stdout:(109/196): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-23T15:40:21.477 INFO:teuthology.orchestra.run.smithi157.stdout:(110/196): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-23T15:40:21.536 INFO:teuthology.orchestra.run.smithi157.stdout:(111/196): python3-urllib3-1.26.5-5.el9.noarch. 3.1 MB/s | 187 kB 00:00 2024-03-23T15:40:21.669 INFO:teuthology.orchestra.run.smithi157.stdout:(112/196): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-23T15:40:21.753 INFO:teuthology.orchestra.run.smithi157.stdout:(113/196): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 803 kB 00:00 2024-03-23T15:40:21.795 INFO:teuthology.orchestra.run.smithi157.stdout:(114/196): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-03-23T15:40:21.837 INFO:teuthology.orchestra.run.smithi157.stdout:(115/196): python3-requests-2.25.1-8.el9.noarch 2.9 MB/s | 125 kB 00:00 2024-03-23T15:40:21.921 INFO:teuthology.orchestra.run.smithi157.stdout:(116/196): elfutils-libelf-devel-0.190-2.el9.x8 258 kB/s | 21 kB 00:00 2024-03-23T15:40:21.996 INFO:teuthology.orchestra.run.smithi157.stdout:(117/196): libuuid-devel-2.37.4-18.el9.x86_64.r 339 kB/s | 25 kB 00:00 2024-03-23T15:40:22.713 INFO:teuthology.orchestra.run.smithi157.stdout:(118/196): rbd-mirror-debuginfo-19.0.0-1855.g21 3.7 MB/s | 51 MB 00:13 2024-03-23T15:40:22.784 INFO:teuthology.orchestra.run.smithi129.stdout:(21/196): ceph-test-19.0.0-1855.g21b3bd67.el9.x 1.6 MB/s | 47 MB 00:30 2024-03-23T15:40:22.909 INFO:teuthology.orchestra.run.smithi129.stdout:(22/196): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 219 kB 00:00 2024-03-23T15:40:22.921 INFO:teuthology.orchestra.run.smithi157.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 1.2 MB/s | 245 kB 00:00 2024-03-23T15:40:22.959 INFO:teuthology.orchestra.run.smithi129.stdout:(23/196): cephfs-shell-19.0.0-1855.g21b3bd67.el 546 kB/s | 27 kB 00:00 2024-03-23T15:40:23.005 INFO:teuthology.orchestra.run.smithi157.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 2.9 MB/s | 248 kB 00:00 2024-03-23T15:40:23.009 INFO:teuthology.orchestra.run.smithi129.stdout:(24/196): libcephfs-devel-19.0.0-1855.g21b3bd67 634 kB/s | 32 kB 00:00 2024-03-23T15:40:23.097 INFO:teuthology.orchestra.run.smithi157.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-23T15:40:23.143 INFO:teuthology.orchestra.run.smithi129.stdout:(25/196): libcephfs2-19.0.0-1855.g21b3bd67.el9. 5.4 MB/s | 744 kB 00:00 2024-03-23T15:40:23.164 INFO:teuthology.orchestra.run.smithi157.stdout:(122/196): qatlib-service-23.11.0-1.el9.x86_64. 524 kB/s | 35 kB 00:00 2024-03-23T15:40:23.223 INFO:teuthology.orchestra.run.smithi157.stdout:(123/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 765 kB/s | 44 kB 00:00 2024-03-23T15:40:23.281 INFO:teuthology.orchestra.run.smithi157.stdout:(124/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-23T15:40:23.357 INFO:teuthology.orchestra.run.smithi157.stdout:(125/196): openssl-devel-3.0.7-27.el9.x86_64.rp 3.0 MB/s | 4.1 MB 00:01 2024-03-23T15:40:23.382 INFO:teuthology.orchestra.run.smithi157.stdout:(126/196): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-03-23T15:40:23.515 INFO:teuthology.orchestra.run.smithi157.stdout:(127/196): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-03-23T15:40:23.749 INFO:teuthology.orchestra.run.smithi157.stdout:(128/196): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-23T15:40:23.810 INFO:teuthology.orchestra.run.smithi157.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-23T15:40:23.850 INFO:teuthology.orchestra.run.smithi157.stdout:(130/196): gperftools-libs-2.9.1-2.el9.x86_64.r 927 kB/s | 309 kB 00:00 2024-03-23T15:40:23.908 INFO:teuthology.orchestra.run.smithi157.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-23T15:40:23.967 INFO:teuthology.orchestra.run.smithi157.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-03-23T15:40:24.001 INFO:teuthology.orchestra.run.smithi129.stdout:(26/196): libcephfs2-debuginfo-19.0.0-1855.g21b 6.3 MB/s | 5.4 MB 00:00 2024-03-23T15:40:24.034 INFO:teuthology.orchestra.run.smithi157.stdout:(133/196): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-23T15:40:24.052 INFO:teuthology.orchestra.run.smithi129.stdout:(27/196): libcephsqlite-19.0.0-1855.g21b3bd67.e 3.2 MB/s | 163 kB 00:00 2024-03-23T15:40:24.059 INFO:teuthology.orchestra.run.smithi157.stdout:(134/196): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-23T15:40:24.093 INFO:teuthology.orchestra.run.smithi157.stdout:(135/196): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-23T15:40:24.102 INFO:teuthology.orchestra.run.smithi129.stdout:(28/196): librados-devel-19.0.0-1855.g21b3bd67. 2.5 MB/s | 127 kB 00:00 2024-03-23T15:40:24.152 INFO:teuthology.orchestra.run.smithi129.stdout:(29/196): librados-devel-debuginfo-19.0.0-1855. 2.1 MB/s | 109 kB 00:00 2024-03-23T15:40:24.160 INFO:teuthology.orchestra.run.smithi157.stdout:(136/196): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-03-23T15:40:24.194 INFO:teuthology.orchestra.run.smithi157.stdout:(137/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-03-23T15:40:24.218 INFO:teuthology.orchestra.run.smithi157.stdout:(138/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 745 kB/s | 43 kB 00:00 2024-03-23T15:40:24.252 INFO:teuthology.orchestra.run.smithi157.stdout:(139/196): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-03-23T15:40:24.277 INFO:teuthology.orchestra.run.smithi157.stdout:(140/196): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-03-23T15:40:24.311 INFO:teuthology.orchestra.run.smithi157.stdout:(141/196): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-23T15:40:24.645 INFO:teuthology.orchestra.run.smithi157.stdout:(142/196): python3-cherrypy-18.6.1-2.el9.noarch 976 kB/s | 358 kB 00:00 2024-03-23T15:40:24.670 INFO:teuthology.orchestra.run.smithi157.stdout:(143/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 845 kB/s | 303 kB 00:00 2024-03-23T15:40:24.745 INFO:teuthology.orchestra.run.smithi157.stdout:(144/196): python3-colorama-0.4.4-12.el9.noarch 353 kB/s | 35 kB 00:00 2024-03-23T15:40:24.771 INFO:teuthology.orchestra.run.smithi157.stdout:(145/196): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-23T15:40:24.830 INFO:teuthology.orchestra.run.smithi157.stdout:(146/196): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-23T15:40:24.896 INFO:teuthology.orchestra.run.smithi157.stdout:(147/196): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-23T15:40:24.921 INFO:teuthology.orchestra.run.smithi157.stdout:(148/196): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-23T15:40:24.947 INFO:teuthology.orchestra.run.smithi157.stdout:(149/196): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-23T15:40:24.972 INFO:teuthology.orchestra.run.smithi157.stdout:(150/196): python3-jaraco-collections-3.0.0-8.e 458 kB/s | 23 kB 00:00 2024-03-23T15:40:25.006 INFO:teuthology.orchestra.run.smithi157.stdout:(151/196): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-03-23T15:40:25.022 INFO:teuthology.orchestra.run.smithi157.stdout:(152/196): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-23T15:40:25.081 INFO:teuthology.orchestra.run.smithi157.stdout:(153/196): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-03-23T15:40:25.098 INFO:teuthology.orchestra.run.smithi157.stdout:(154/196): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-03-23T15:40:25.148 INFO:teuthology.orchestra.run.smithi157.stdout:(155/196): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-23T15:40:25.174 INFO:teuthology.orchestra.run.smithi157.stdout:(156/196): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-03-23T15:40:25.207 INFO:teuthology.orchestra.run.smithi157.stdout:(157/196): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-23T15:40:25.258 INFO:teuthology.orchestra.run.smithi157.stdout:(158/196): python3-portend-3.1.0-2.el9.noarch.r 195 kB/s | 16 kB 00:00 2024-03-23T15:40:25.283 INFO:teuthology.orchestra.run.smithi157.stdout:(159/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-23T15:40:25.317 INFO:teuthology.orchestra.run.smithi157.stdout:(160/196): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-23T15:40:25.342 INFO:teuthology.orchestra.run.smithi157.stdout:(161/196): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-03-23T15:40:25.376 INFO:teuthology.orchestra.run.smithi157.stdout:(162/196): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-23T15:40:25.401 INFO:teuthology.orchestra.run.smithi157.stdout:(163/196): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-23T15:40:25.435 INFO:teuthology.orchestra.run.smithi157.stdout:(164/196): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-03-23T15:40:25.460 INFO:teuthology.orchestra.run.smithi157.stdout:(165/196): python3-typing-extensions-4.4.0-2.el 871 kB/s | 51 kB 00:00 2024-03-23T15:40:25.494 INFO:teuthology.orchestra.run.smithi157.stdout:(166/196): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-23T15:40:25.519 INFO:teuthology.orchestra.run.smithi157.stdout:(167/196): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-23T15:40:25.569 INFO:teuthology.orchestra.run.smithi157.stdout:(168/196): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-23T15:40:25.594 INFO:teuthology.orchestra.run.smithi157.stdout:(169/196): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-03-23T15:40:25.628 INFO:teuthology.orchestra.run.smithi157.stdout:(170/196): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-23T15:40:25.678 INFO:teuthology.orchestra.run.smithi157.stdout:(171/196): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-23T15:40:26.370 INFO:teuthology.orchestra.run.smithi157.stdout:(172/196): librbd1-19.0.0-1855.g21b3bd67.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-03-23T15:40:26.429 INFO:teuthology.orchestra.run.smithi157.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-23T15:40:26.621 INFO:teuthology.orchestra.run.smithi157.stdout:(174/196): NetworkManager-1.47.2-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-23T15:40:26.704 INFO:teuthology.orchestra.run.smithi157.stdout:(175/196): NetworkManager-libnm-1.47.2-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-23T15:40:26.746 INFO:teuthology.orchestra.run.smithi157.stdout:(176/196): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-23T15:40:26.788 INFO:teuthology.orchestra.run.smithi157.stdout:(177/196): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-23T15:40:26.830 INFO:teuthology.orchestra.run.smithi157.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-23T15:40:26.872 INFO:teuthology.orchestra.run.smithi157.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 880 kB/s | 36 kB 00:00 2024-03-23T15:40:26.921 INFO:teuthology.orchestra.run.smithi157.stdout:(180/196): elfutils-libelf-0.190-2.el9.x86_64.r 3.9 MB/s | 192 kB 00:00 2024-03-23T15:40:26.963 INFO:teuthology.orchestra.run.smithi157.stdout:(181/196): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-23T15:40:27.005 INFO:teuthology.orchestra.run.smithi157.stdout:(182/196): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-03-23T15:40:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:(183/196): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-23T15:40:27.089 INFO:teuthology.orchestra.run.smithi157.stdout:(184/196): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-23T15:40:27.131 INFO:teuthology.orchestra.run.smithi157.stdout:(185/196): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-23T15:40:27.173 INFO:teuthology.orchestra.run.smithi157.stdout:(186/196): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-23T15:40:27.215 INFO:teuthology.orchestra.run.smithi157.stdout:(187/196): libuuid-2.37.4-18.el9.x86_64.rpm 689 kB/s | 29 kB 00:00 2024-03-23T15:40:27.274 INFO:teuthology.orchestra.run.smithi157.stdout:(188/196): librados2-19.0.0-1855.g21b3bd67.el9. 2.1 MB/s | 3.4 MB 00:01 2024-03-23T15:40:27.316 INFO:teuthology.orchestra.run.smithi157.stdout:(189/196): openssl-3.0.7-27.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2024-03-23T15:40:27.391 INFO:teuthology.orchestra.run.smithi157.stdout:(190/196): util-linux-2.37.4-18.el9.x86_64.rpm 30 MB/s | 2.3 MB 00:00 2024-03-23T15:40:27.433 INFO:teuthology.orchestra.run.smithi157.stdout:(191/196): util-linux-core-2.37.4-18.el9.x86_64 11 MB/s | 465 kB 00:00 2024-03-23T15:40:27.475 INFO:teuthology.orchestra.run.smithi157.stdout:(192/196): util-linux-user-2.37.4-18.el9.x86_64 745 kB/s | 31 kB 00:00 2024-03-23T15:40:27.525 INFO:teuthology.orchestra.run.smithi157.stdout:(193/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 22 MB/s | 1.1 MB 00:00 2024-03-23T15:40:27.576 INFO:teuthology.orchestra.run.smithi157.stdout:(194/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 2.2 MB 00:00 2024-03-23T15:40:27.651 INFO:teuthology.orchestra.run.smithi157.stdout:(195/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 593 kB/s | 74 kB 00:00 2024-03-23T15:40:30.469 INFO:teuthology.orchestra.run.smithi129.stdout:(30/196): librados2-debuginfo-19.0.0-1855.g21b3 7.0 MB/s | 44 MB 00:06 2024-03-23T15:40:30.602 INFO:teuthology.orchestra.run.smithi129.stdout:(31/196): libradosstriper1-19.0.0-1855.g21b3bd6 3.7 MB/s | 499 kB 00:00 2024-03-23T15:40:35.002 INFO:teuthology.orchestra.run.smithi129.stdout:(32/196): librbd1-debuginfo-19.0.0-1855.g21b3bd 12 MB/s | 52 MB 00:04 2024-03-23T15:40:35.453 INFO:teuthology.orchestra.run.smithi129.stdout:(33/196): librgw2-19.0.0-1855.g21b3bd67.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-23T15:40:35.519 INFO:teuthology.orchestra.run.smithi129.stdout:(34/196): python3-ceph-argparse-19.0.0-1855.g21 679 kB/s | 45 kB 00:00 2024-03-23T15:40:35.586 INFO:teuthology.orchestra.run.smithi129.stdout:(35/196): python3-ceph-common-19.0.0-1855.g21b3 1.9 MB/s | 131 kB 00:00 2024-03-23T15:40:35.661 INFO:teuthology.orchestra.run.smithi129.stdout:(36/196): python3-cephfs-19.0.0-1855.g21b3bd67. 2.1 MB/s | 161 kB 00:00 2024-03-23T15:40:35.754 INFO:teuthology.orchestra.run.smithi129.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-1855. 6.5 MB/s | 615 kB 00:00 2024-03-23T15:40:35.829 INFO:teuthology.orchestra.run.smithi129.stdout:(38/196): python3-rados-19.0.0-1855.g21b3bd67.e 4.2 MB/s | 320 kB 00:00 2024-03-23T15:40:35.929 INFO:teuthology.orchestra.run.smithi129.stdout:(39/196): python3-rados-debuginfo-19.0.0-1855.g 7.5 MB/s | 770 kB 00:00 2024-03-23T15:40:36.005 INFO:teuthology.orchestra.run.smithi129.stdout:(40/196): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.9 MB/s | 298 kB 00:00 2024-03-23T15:40:36.147 INFO:teuthology.orchestra.run.smithi129.stdout:(41/196): python3-rbd-debuginfo-19.0.0-1855.g21 8.0 MB/s | 1.1 MB 00:00 2024-03-23T15:40:36.205 INFO:teuthology.orchestra.run.smithi129.stdout:(42/196): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.7 MB/s | 99 kB 00:00 2024-03-23T15:40:36.280 INFO:teuthology.orchestra.run.smithi129.stdout:(43/196): python3-rgw-debuginfo-19.0.0-1855.g21 4.4 MB/s | 337 kB 00:00 2024-03-23T15:40:36.347 INFO:teuthology.orchestra.run.smithi129.stdout:(44/196): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-23T15:40:36.431 INFO:teuthology.orchestra.run.smithi129.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-1855.g21b3b 6.0 MB/s | 509 kB 00:00 2024-03-23T15:40:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:(46/196): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 11 MB/s | 3.1 MB 00:00 2024-03-23T15:40:40.715 INFO:teuthology.orchestra.run.smithi129.stdout:(47/196): rbd-mirror-debuginfo-19.0.0-1855.g21b 13 MB/s | 51 MB 00:04 2024-03-23T15:40:40.765 INFO:teuthology.orchestra.run.smithi129.stdout:(48/196): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-23T15:40:40.957 INFO:teuthology.orchestra.run.smithi129.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-1855.g21b3bd 6.5 MB/s | 1.3 MB 00:00 2024-03-23T15:40:41.015 INFO:teuthology.orchestra.run.smithi129.stdout:(50/196): ceph-grafana-dashboards-19.0.0-1855.g 541 kB/s | 31 kB 00:00 2024-03-23T15:40:41.091 INFO:teuthology.orchestra.run.smithi129.stdout:(51/196): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.8 MB/s | 138 kB 00:00 2024-03-23T15:40:41.283 INFO:teuthology.orchestra.run.smithi129.stdout:(52/196): ceph-mgr-dashboard-19.0.0-1855.g21b3b 9.3 MB/s | 1.8 MB 00:00 2024-03-23T15:40:41.958 INFO:teuthology.orchestra.run.smithi129.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-23T15:40:42.033 INFO:teuthology.orchestra.run.smithi129.stdout:(54/196): ceph-mgr-modules-core-19.0.0-1855.g21 3.3 MB/s | 248 kB 00:00 2024-03-23T15:40:42.092 INFO:teuthology.orchestra.run.smithi129.stdout:(55/196): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 847 kB/s | 49 kB 00:00 2024-03-23T15:40:42.150 INFO:teuthology.orchestra.run.smithi129.stdout:(56/196): ceph-prometheus-alerts-19.0.0-1855.g2 263 kB/s | 15 kB 00:00 2024-03-23T15:40:42.226 INFO:teuthology.orchestra.run.smithi129.stdout:(57/196): ceph-volume-19.0.0-1855.g21b3bd67.el9 3.4 MB/s | 264 kB 00:00 2024-03-23T15:40:42.326 INFO:teuthology.orchestra.run.smithi129.stdout:(58/196): cephadm-19.0.0-1855.g21b3bd67.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-23T15:40:42.385 INFO:teuthology.orchestra.run.smithi129.stdout:(59/196): cephfs-top-19.0.0-1855.g21b3bd67.el9. 312 kB/s | 18 kB 00:00 2024-03-23T15:40:42.677 INFO:teuthology.orchestra.run.smithi129.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-23T15:40:42.777 INFO:teuthology.orchestra.run.smithi129.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 9.0 MB/s | 920 kB 00:00 2024-03-23T15:40:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-23T15:40:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-03-23T15:40:43.027 INFO:teuthology.orchestra.run.smithi129.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2024-03-23T15:40:43.203 INFO:teuthology.orchestra.run.smithi129.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-23T15:40:43.245 INFO:teuthology.orchestra.run.smithi129.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 412 kB/s | 17 kB 00:00 2024-03-23T15:40:43.279 INFO:teuthology.orchestra.run.smithi129.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-23T15:40:43.312 INFO:teuthology.orchestra.run.smithi129.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-23T15:40:43.338 INFO:teuthology.orchestra.run.smithi129.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-23T15:40:43.388 INFO:teuthology.orchestra.run.smithi129.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-23T15:40:43.447 INFO:teuthology.orchestra.run.smithi129.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 578 kB 00:00 2024-03-23T15:40:43.488 INFO:teuthology.orchestra.run.smithi129.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-23T15:40:43.555 INFO:teuthology.orchestra.run.smithi129.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-23T15:40:43.622 INFO:teuthology.orchestra.run.smithi129.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-23T15:40:43.690 INFO:teuthology.orchestra.run.smithi129.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-23T15:40:43.749 INFO:teuthology.orchestra.run.smithi129.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-23T15:40:43.957 INFO:teuthology.orchestra.run.smithi129.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-03-23T15:40:44.032 INFO:teuthology.orchestra.run.smithi129.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-23T15:40:44.091 INFO:teuthology.orchestra.run.smithi129.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-23T15:40:44.132 INFO:teuthology.orchestra.run.smithi129.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-23T15:40:44.366 INFO:teuthology.orchestra.run.smithi129.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 24 MB/s | 5.7 MB 00:00 2024-03-23T15:40:44.442 INFO:teuthology.orchestra.run.smithi129.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-23T15:40:44.492 INFO:teuthology.orchestra.run.smithi129.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 78 kB 00:00 2024-03-23T15:40:44.734 INFO:teuthology.orchestra.run.smithi129.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2024-03-23T15:40:44.809 INFO:teuthology.orchestra.run.smithi129.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 326 kB/s | 24 kB 00:00 2024-03-23T15:40:44.876 INFO:teuthology.orchestra.run.smithi129.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 272 kB/s | 18 kB 00:00 2024-03-23T15:40:44.951 INFO:teuthology.orchestra.run.smithi129.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-23T15:40:45.010 INFO:teuthology.orchestra.run.smithi129.stdout:(88/196): python3-markupsafe-1.1.1-12.el9.x86_6 540 kB/s | 32 kB 00:00 2024-03-23T15:40:45.210 INFO:teuthology.orchestra.run.smithi129.stdout:(89/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-23T15:40:45.269 INFO:teuthology.orchestra.run.smithi129.stdout:(90/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 217 kB 00:00 2024-03-23T15:40:45.327 INFO:teuthology.orchestra.run.smithi129.stdout:(91/196): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-23T15:40:45.386 INFO:teuthology.orchestra.run.smithi129.stdout:(92/196): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-23T15:40:45.445 INFO:teuthology.orchestra.run.smithi129.stdout:(93/196): python3-prettytable-0.7.2-27.el9.noar 701 kB/s | 41 kB 00:00 2024-03-23T15:40:45.495 INFO:teuthology.orchestra.run.smithi129.stdout:(94/196): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-23T15:40:45.537 INFO:teuthology.orchestra.run.smithi129.stdout:(95/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-23T15:40:45.587 INFO:teuthology.orchestra.run.smithi129.stdout:(96/196): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-23T15:40:45.638 INFO:teuthology.orchestra.run.smithi129.stdout:(97/196): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-23T15:40:45.671 INFO:teuthology.orchestra.run.smithi129.stdout:(98/196): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-23T15:40:46.263 INFO:teuthology.orchestra.run.smithi129.stdout:(99/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-23T15:40:46.297 INFO:teuthology.orchestra.run.smithi129.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-23T15:40:46.330 INFO:teuthology.orchestra.run.smithi129.stdout:(101/196): python3-wcwidth-0.2.5-8.el9.noarch.r 1.2 MB/s | 41 kB 00:00 2024-03-23T15:40:46.372 INFO:teuthology.orchestra.run.smithi129.stdout:(102/196): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-23T15:40:46.406 INFO:teuthology.orchestra.run.smithi129.stdout:(103/196): uuid-1.6.2-55.el9.x86_64.rpm 1.6 MB/s | 56 kB 00:00 2024-03-23T15:40:46.440 INFO:teuthology.orchestra.run.smithi129.stdout:(104/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-23T15:40:46.473 INFO:teuthology.orchestra.run.smithi129.stdout:(105/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-23T15:40:46.507 INFO:teuthology.orchestra.run.smithi129.stdout:(106/196): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-23T15:40:46.549 INFO:teuthology.orchestra.run.smithi129.stdout:(107/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-23T15:40:46.607 INFO:teuthology.orchestra.run.smithi129.stdout:(108/196): python3-cryptography-36.0.1-4.el9.x8 20 MB/s | 1.1 MB 00:00 2024-03-23T15:40:46.641 INFO:teuthology.orchestra.run.smithi129.stdout:(109/196): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-23T15:40:46.675 INFO:teuthology.orchestra.run.smithi129.stdout:(110/196): python3-pycparser-2.20-6.el9.noarch. 3.6 MB/s | 124 kB 00:00 2024-03-23T15:40:46.717 INFO:teuthology.orchestra.run.smithi129.stdout:(111/196): python3-urllib3-1.26.5-5.el9.noarch. 4.4 MB/s | 187 kB 00:00 2024-03-23T15:40:46.851 INFO:teuthology.orchestra.run.smithi129.stdout:(112/196): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-23T15:40:46.934 INFO:teuthology.orchestra.run.smithi129.stdout:(113/196): libgfortran-11.4.1-3.el9.x86_64.rpm 9.5 MB/s | 803 kB 00:00 2024-03-23T15:40:46.976 INFO:teuthology.orchestra.run.smithi129.stdout:(114/196): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-03-23T15:40:47.018 INFO:teuthology.orchestra.run.smithi129.stdout:(115/196): python3-requests-2.25.1-8.el9.noarch 2.9 MB/s | 125 kB 00:00 2024-03-23T15:40:47.052 INFO:teuthology.orchestra.run.smithi129.stdout:(116/196): elfutils-libelf-devel-0.190-2.el9.x8 645 kB/s | 21 kB 00:00 2024-03-23T15:40:47.085 INFO:teuthology.orchestra.run.smithi129.stdout:(117/196): libuuid-devel-2.37.4-18.el9.x86_64.r 767 kB/s | 25 kB 00:00 2024-03-23T15:40:47.219 INFO:teuthology.orchestra.run.smithi129.stdout:(118/196): openssl-devel-3.0.7-27.el9.x86_64.rp 31 MB/s | 4.1 MB 00:00 2024-03-23T15:40:47.261 INFO:teuthology.orchestra.run.smithi129.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 5.8 MB/s | 245 kB 00:00 2024-03-23T15:40:47.303 INFO:teuthology.orchestra.run.smithi129.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 5.8 MB/s | 248 kB 00:00 2024-03-23T15:40:47.345 INFO:teuthology.orchestra.run.smithi129.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 5.0 MB/s | 216 kB 00:00 2024-03-23T15:40:47.379 INFO:teuthology.orchestra.run.smithi129.stdout:(122/196): qatlib-service-23.11.0-1.el9.x86_64. 1.0 MB/s | 35 kB 00:00 2024-03-23T15:40:47.412 INFO:teuthology.orchestra.run.smithi129.stdout:(123/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-23T15:40:47.454 INFO:teuthology.orchestra.run.smithi129.stdout:(124/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-23T15:40:47.496 INFO:teuthology.orchestra.run.smithi129.stdout:(125/196): lua-devel-5.4.4-4.el9.x86_64.rpm 537 kB/s | 22 kB 00:00 2024-03-23T15:40:47.630 INFO:teuthology.orchestra.run.smithi129.stdout:(126/196): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-23T15:40:47.814 INFO:teuthology.orchestra.run.smithi129.stdout:(127/196): abseil-cpp-20211102.0-3.el9.x86_64.r 2.9 MB/s | 548 kB 00:00 2024-03-23T15:40:47.864 INFO:teuthology.orchestra.run.smithi129.stdout:(128/196): gperftools-libs-2.9.1-2.el9.x86_64.r 6.1 MB/s | 309 kB 00:00 2024-03-23T15:40:47.906 INFO:teuthology.orchestra.run.smithi129.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-23T15:40:48.056 INFO:teuthology.orchestra.run.smithi129.stdout:(130/196): libarrow-9.0.0-10.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-03-23T15:40:48.098 INFO:teuthology.orchestra.run.smithi129.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-03-23T15:40:48.140 INFO:teuthology.orchestra.run.smithi129.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-23T15:40:48.182 INFO:teuthology.orchestra.run.smithi129.stdout:(133/196): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-23T15:40:48.224 INFO:teuthology.orchestra.run.smithi129.stdout:(134/196): luarocks-3.9.2-1.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-03-23T15:40:48.274 INFO:teuthology.orchestra.run.smithi129.stdout:(135/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-03-23T15:40:48.333 INFO:teuthology.orchestra.run.smithi129.stdout:(136/196): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-03-23T15:40:48.374 INFO:teuthology.orchestra.run.smithi129.stdout:(137/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-03-23T15:40:48.416 INFO:teuthology.orchestra.run.smithi129.stdout:(138/196): python3-cachetools-4.2.4-1.el9.noarc 772 kB/s | 32 kB 00:00 2024-03-23T15:40:48.458 INFO:teuthology.orchestra.run.smithi129.stdout:(139/196): python3-certifi-2021.10.8-2.el9.noar 362 kB/s | 15 kB 00:00 2024-03-23T15:40:48.501 INFO:teuthology.orchestra.run.smithi129.stdout:(140/196): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-03-23T15:40:48.542 INFO:teuthology.orchestra.run.smithi129.stdout:(141/196): python3-cherrypy-18.6.1-2.el9.noarch 8.4 MB/s | 358 kB 00:00 2024-03-23T15:40:48.584 INFO:teuthology.orchestra.run.smithi129.stdout:(142/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-03-23T15:40:48.618 INFO:teuthology.orchestra.run.smithi129.stdout:(143/196): python3-colorama-0.4.4-12.el9.noarch 1.0 MB/s | 35 kB 00:00 2024-03-23T15:40:48.660 INFO:teuthology.orchestra.run.smithi129.stdout:(144/196): python3-google-auth-2.28.2-1.el9.noa 4.8 MB/s | 203 kB 00:00 2024-03-23T15:40:48.727 INFO:teuthology.orchestra.run.smithi129.stdout:(145/196): python3-grpcio-1.46.7-10.el9.x86_64. 31 MB/s | 2.0 MB 00:00 2024-03-23T15:40:48.769 INFO:teuthology.orchestra.run.smithi129.stdout:(146/196): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-03-23T15:40:48.802 INFO:teuthology.orchestra.run.smithi129.stdout:(147/196): python3-jaraco-8.2.1-3.el9.noarch.rp 323 kB/s | 11 kB 00:00 2024-03-23T15:40:48.836 INFO:teuthology.orchestra.run.smithi129.stdout:(148/196): python3-jaraco-classes-3.2.1-5.el9.n 532 kB/s | 18 kB 00:00 2024-03-23T15:40:48.870 INFO:teuthology.orchestra.run.smithi129.stdout:(149/196): python3-jaraco-collections-3.0.0-8.e 691 kB/s | 23 kB 00:00 2024-03-23T15:40:48.903 INFO:teuthology.orchestra.run.smithi129.stdout:(150/196): python3-jaraco-functools-3.5.0-2.el9 586 kB/s | 19 kB 00:00 2024-03-23T15:40:48.937 INFO:teuthology.orchestra.run.smithi129.stdout:(151/196): python3-jaraco-text-3.2.0-6.el9.noar 592 kB/s | 20 kB 00:00 2024-03-23T15:40:48.996 INFO:teuthology.orchestra.run.smithi129.stdout:(152/196): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-03-23T15:40:49.029 INFO:teuthology.orchestra.run.smithi129.stdout:(153/196): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-23T15:40:49.071 INFO:teuthology.orchestra.run.smithi129.stdout:(154/196): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-03-23T15:40:49.105 INFO:teuthology.orchestra.run.smithi129.stdout:(155/196): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-03-23T15:40:49.147 INFO:teuthology.orchestra.run.smithi129.stdout:(156/196): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-03-23T15:40:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:(157/196): python3-portend-3.1.0-2.el9.noarch.r 493 kB/s | 16 kB 00:00 2024-03-23T15:40:49.222 INFO:teuthology.orchestra.run.smithi129.stdout:(158/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-03-23T15:40:49.264 INFO:teuthology.orchestra.run.smithi129.stdout:(159/196): python3-pyperclip-1.8.2-2.el9.noarch 678 kB/s | 28 kB 00:00 2024-03-23T15:40:49.306 INFO:teuthology.orchestra.run.smithi129.stdout:(160/196): python3-repoze-lru-0.7-16.el9.noarch 742 kB/s | 31 kB 00:00 2024-03-23T15:40:49.348 INFO:teuthology.orchestra.run.smithi129.stdout:(161/196): python3-routes-2.5.1-5.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2024-03-23T15:40:49.391 INFO:teuthology.orchestra.run.smithi129.stdout:(162/196): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-23T15:40:49.433 INFO:teuthology.orchestra.run.smithi129.stdout:(163/196): python3-tempora-5.0.0-2.el9.noarch.r 862 kB/s | 36 kB 00:00 2024-03-23T15:40:49.474 INFO:teuthology.orchestra.run.smithi129.stdout:(164/196): python3-typing-extensions-4.4.0-2.el 1.2 MB/s | 51 kB 00:00 2024-03-23T15:40:49.516 INFO:teuthology.orchestra.run.smithi129.stdout:(165/196): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-03-23T15:40:49.558 INFO:teuthology.orchestra.run.smithi129.stdout:(166/196): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-23T15:40:49.609 INFO:teuthology.orchestra.run.smithi129.stdout:(167/196): python3-werkzeug-2.0.3-3.el9.1.noarc 8.4 MB/s | 427 kB 00:00 2024-03-23T15:40:49.650 INFO:teuthology.orchestra.run.smithi129.stdout:(168/196): python3-zc-lockfile-2.0-10.el9.noarc 482 kB/s | 20 kB 00:00 2024-03-23T15:40:49.709 INFO:teuthology.orchestra.run.smithi129.stdout:(169/196): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-23T15:40:49.768 INFO:teuthology.orchestra.run.smithi129.stdout:(170/196): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-23T15:40:50.635 INFO:teuthology.orchestra.run.smithi129.stdout:(171/196): librados2-19.0.0-1855.g21b3bd67.el9. 3.9 MB/s | 3.4 MB 00:00 2024-03-23T15:40:51.168 INFO:teuthology.orchestra.run.smithi129.stdout:(172/196): librbd1-19.0.0-1855.g21b3bd67.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-03-23T15:40:51.210 INFO:teuthology.orchestra.run.smithi129.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-23T15:40:51.396 INFO:teuthology.orchestra.run.smithi129.stdout:(174/196): NetworkManager-1.47.2-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-23T15:40:51.463 INFO:teuthology.orchestra.run.smithi129.stdout:(175/196): NetworkManager-libnm-1.47.2-1.el9.x8 28 MB/s | 1.8 MB 00:00 2024-03-23T15:40:51.505 INFO:teuthology.orchestra.run.smithi129.stdout:(176/196): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-23T15:40:51.548 INFO:teuthology.orchestra.run.smithi129.stdout:(177/196): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-23T15:40:51.590 INFO:teuthology.orchestra.run.smithi129.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-23T15:40:51.623 INFO:teuthology.orchestra.run.smithi129.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 36 kB 00:00 2024-03-23T15:40:51.665 INFO:teuthology.orchestra.run.smithi129.stdout:(180/196): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 192 kB 00:00 2024-03-23T15:40:51.707 INFO:teuthology.orchestra.run.smithi129.stdout:(181/196): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-23T15:40:51.749 INFO:teuthology.orchestra.run.smithi129.stdout:(182/196): libacl-2.3.1-4.el9.x86_64.rpm 552 kB/s | 23 kB 00:00 2024-03-23T15:40:51.791 INFO:teuthology.orchestra.run.smithi129.stdout:(183/196): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-23T15:40:52.063 INFO:teuthology.orchestra.run.smithi129.stdout:(184/196): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-23T15:40:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:(185/196): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-23T15:40:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:(186/196): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-23T15:40:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:(187/196): libuuid-2.37.4-18.el9.x86_64.rpm 492 kB/s | 29 kB 00:00 2024-03-23T15:40:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:(188/196): openssl-3.0.7-27.el9.x86_64.rpm 24 MB/s | 1.2 MB 00:00 2024-03-23T15:40:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:(189/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 32 MB/s | 2.2 MB 00:00 2024-03-23T15:40:52.160 INFO:teuthology.orchestra.run.smithi129.stdout:(190/196): util-linux-2.37.4-18.el9.x86_64.rpm 34 MB/s | 2.3 MB 00:00 2024-03-23T15:40:52.202 INFO:teuthology.orchestra.run.smithi129.stdout:(191/196): util-linux-core-2.37.4-18.el9.x86_64 11 MB/s | 465 kB 00:00 2024-03-23T15:40:52.244 INFO:teuthology.orchestra.run.smithi129.stdout:(192/196): util-linux-user-2.37.4-18.el9.x86_64 751 kB/s | 31 kB 00:00 2024-03-23T15:40:52.302 INFO:teuthology.orchestra.run.smithi129.stdout:(193/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-23T15:40:52.345 INFO:teuthology.orchestra.run.smithi129.stdout:(194/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 74 kB 00:00 2024-03-23T15:41:07.710 INFO:teuthology.orchestra.run.smithi129.stdout:(195/196): ceph-radosgw-debuginfo-19.0.0-1855.g 2.0 MB/s | 176 MB 01:30 2024-03-23T15:41:09.021 INFO:teuthology.orchestra.run.smithi157.stdout:(196/196): ceph-test-debuginfo-19.0.0-1855.g21b 7.2 MB/s | 674 MB 01:34 2024-03-23T15:41:09.028 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-23T15:41:09.028 INFO:teuthology.orchestra.run.smithi157.stdout:Total 11 MB/s | 2.2 GB 03:16 2024-03-23T15:41:09.476 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-23T15:41:09.760 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-23T15:41:09.761 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-23T15:41:15.684 INFO:teuthology.orchestra.run.smithi129.stdout:(196/196): ceph-test-debuginfo-19.0.0-1855.g21b 9.9 MB/s | 674 MB 01:08 2024-03-23T15:41:15.687 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-23T15:41:15.688 INFO:teuthology.orchestra.run.smithi129.stdout:Total 12 MB/s | 2.2 GB 03:08 2024-03-23T15:41:16.124 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-23T15:41:16.404 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-23T15:41:16.404 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-23T15:41:23.492 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-23T15:41:23.492 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-23T15:41:30.165 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-23T15:41:30.166 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-23T15:41:42.680 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-23T15:41:45.181 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-23T15:41:45.322 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 2/220 2024-03-23T15:41:45.348 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-23T15:41:45.385 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-23T15:41:45.395 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-23T15:41:45.446 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-23T15:41:45.498 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-23T15:41:45.531 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-23T15:41:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-23T15:41:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-23T15:41:45.929 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-23T15:41:45.985 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-23T15:41:46.106 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-23T15:41:46.136 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-23T15:41:46.227 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-23T15:41:46.266 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-23T15:41:46.304 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-23T15:41:46.490 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-23T15:41:46.529 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-23T15:41:46.549 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-23T15:41:46.646 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-23T15:41:46.669 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-23T15:41:46.734 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-23T15:41:46.773 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-23T15:41:46.795 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-23T15:41:46.811 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-23T15:41:47.004 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-23T15:41:47.035 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-23T15:41:47.067 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-23T15:41:47.262 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-23T15:41:47.394 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-23T15:41:47.462 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-23T15:41:47.558 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-23T15:41:47.605 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-23T15:41:47.828 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-23T15:41:47.936 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-23T15:41:48.030 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-23T15:41:48.060 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-23T15:41:48.087 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-23T15:41:48.122 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-23T15:41:48.196 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-23T15:41:48.250 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-23T15:41:48.288 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-23T15:41:48.361 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-23T15:41:48.396 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-23T15:41:48.439 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-23T15:41:48.473 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-23T15:41:48.500 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-23T15:41:48.569 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-23T15:41:48.605 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 40/220 2024-03-23T15:41:48.648 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 41/220 2024-03-23T15:41:48.677 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 42/220 2024-03-23T15:41:48.746 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-23T15:41:48.791 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-23T15:41:48.865 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-23T15:41:48.998 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-23T15:41:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-23T15:41:49.366 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-23T15:41:49.580 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-23T15:41:49.657 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-23T15:41:49.733 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-23T15:41:49.775 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-23T15:41:49.783 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-23T15:41:49.817 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-23T15:41:49.856 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-23T15:41:49.894 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 54/220 2024-03-23T15:41:49.917 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-23T15:41:50.193 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-23T15:41:50.376 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-23T15:41:50.451 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-23T15:41:50.496 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-23T15:41:50.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-23T15:41:50.673 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-23T15:41:50.729 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-23T15:41:50.818 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-23T15:41:50.857 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-23T15:41:50.893 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-23T15:41:50.941 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-23T15:41:50.976 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-23T15:41:51.028 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-23T15:41:51.070 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-23T15:41:51.206 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-23T15:41:51.254 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-23T15:41:51.281 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-23T15:41:51.322 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-23T15:41:51.425 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-23T15:41:51.480 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 75/220 2024-03-23T15:41:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-23T15:41:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-23T15:41:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-23T15:41:51.624 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-23T15:41:51.764 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-23T15:41:51.862 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-23T15:41:51.898 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-23T15:41:51.904 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-23T15:41:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 2/220 2024-03-23T15:41:51.953 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-23T15:41:51.954 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-23T15:41:52.000 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-23T15:41:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-23T15:41:52.009 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-23T15:41:52.050 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-23T15:41:52.103 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-23T15:41:52.136 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-23T15:41:52.162 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-23T15:41:52.193 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-23T15:41:52.206 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-23T15:41:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-23T15:41:52.378 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-23T15:41:52.415 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 88/220 2024-03-23T15:41:52.447 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-23T15:41:52.477 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-23T15:41:52.505 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-23T15:41:52.539 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-23T15:41:52.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-23T15:41:52.593 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-23T15:41:52.611 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-23T15:41:52.638 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-23T15:41:52.649 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-23T15:41:52.735 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-23T15:41:52.758 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-23T15:41:52.768 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-23T15:41:52.781 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-23T15:41:52.801 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-23T15:41:52.834 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-23T15:41:52.869 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-23T15:41:52.872 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-23T15:41:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-23T15:41:52.952 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-23T15:41:52.958 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-23T15:41:53.048 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-23T15:41:53.093 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-23T15:41:53.123 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-23T15:41:53.153 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-23T15:41:53.170 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-23T15:41:53.199 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-23T15:41:53.228 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-23T15:41:53.259 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-23T15:41:53.308 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-23T15:41:53.315 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-23T15:41:53.337 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-23T15:41:53.394 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-23T15:41:53.433 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-23T15:41:53.455 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-23T15:41:53.470 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-23T15:41:53.664 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-23T15:41:53.695 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-23T15:41:53.727 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-23T15:41:53.984 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-23T15:41:54.036 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-23T15:41:54.062 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-23T15:41:54.080 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-23T15:41:54.113 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-23T15:41:54.123 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-23T15:41:54.162 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-23T15:41:54.236 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-23T15:41:54.251 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-23T15:41:54.281 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-23T15:41:54.290 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-23T15:41:54.459 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-23T15:41:54.503 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-23T15:41:54.506 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-23T15:41:54.581 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-23T15:41:54.615 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-23T15:41:54.708 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-23T15:41:54.746 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-23T15:41:54.772 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-23T15:41:54.808 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-23T15:41:54.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-23T15:41:54.936 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-23T15:41:54.982 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-23T15:41:55.046 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-23T15:41:55.081 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-23T15:41:55.124 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-23T15:41:55.157 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-23T15:41:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-23T15:41:55.224 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-23T15:41:55.252 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-23T15:41:55.318 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 40/220 2024-03-23T15:41:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-23T15:41:55.358 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 41/220 2024-03-23T15:41:55.387 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 42/220 2024-03-23T15:41:55.454 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-23T15:41:55.509 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-23T15:41:55.582 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-23T15:41:55.714 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-23T15:41:55.762 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-23T15:41:56.273 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-23T15:41:56.356 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-23T15:41:56.442 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-23T15:41:56.488 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-23T15:41:56.496 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-23T15:41:56.529 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-23T15:41:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-23T15:41:56.569 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-23T15:41:56.606 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 54/220 2024-03-23T15:41:56.637 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-23T15:41:56.662 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-23T15:41:56.693 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-23T15:41:56.724 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-23T15:41:56.760 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-23T15:41:56.918 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-23T15:41:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-23T15:41:57.099 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-23T15:41:57.127 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-23T15:41:57.189 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-23T15:41:57.198 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-23T15:41:57.220 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 126/220 2024-03-23T15:41:57.242 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-23T15:41:57.256 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-23T15:41:57.386 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-23T15:41:57.445 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-23T15:41:57.500 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-23T15:41:57.572 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-23T15:41:57.604 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-23T15:41:57.640 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-23T15:41:57.688 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-23T15:41:57.700 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-23T15:41:57.723 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-23T15:41:57.735 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-23T15:41:57.766 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-23T15:41:57.790 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-23T15:41:57.808 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-23T15:41:57.811 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/220 2024-03-23T15:41:57.948 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-23T15:41:58.157 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-23T15:41:58.270 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-23T15:41:58.311 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-23T15:41:58.405 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-23T15:41:58.470 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 75/220 2024-03-23T15:41:58.514 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-23T15:41:58.544 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-23T15:41:58.569 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-23T15:41:58.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-23T15:41:58.778 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-23T15:41:58.885 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-23T15:41:58.944 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-23T15:41:58.992 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-23T15:41:59.048 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-23T15:41:59.233 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-23T15:41:59.282 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-23T15:41:59.410 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-23T15:41:59.448 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 88/220 2024-03-23T15:41:59.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-23T15:41:59.510 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-23T15:41:59.538 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-23T15:41:59.572 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-23T15:41:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-23T15:41:59.643 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-23T15:41:59.679 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-23T15:41:59.755 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-23T15:41:59.790 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-23T15:41:59.823 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-23T15:41:59.865 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-23T15:41:59.899 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-23T15:41:59.988 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-23T15:42:00.070 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-23T15:42:00.074 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-23T15:42:00.116 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-23T15:42:00.123 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-23T15:42:00.154 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-23T15:42:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-23T15:42:00.306 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-23T15:42:00.364 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-23T15:42:00.650 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-23T15:42:00.686 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-23T15:42:00.775 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-23T15:42:00.775 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T15:42:00.776 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T15:42:00.776 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:00.812 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-23T15:42:01.082 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-23T15:42:01.128 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-23T15:42:01.162 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-23T15:42:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-23T15:42:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-23T15:42:01.371 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-23T15:42:01.548 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-23T15:42:01.577 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-23T15:42:01.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-23T15:42:02.288 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-23T15:42:02.391 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-23T15:42:03.610 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-23T15:42:03.713 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-23T15:42:03.744 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-23T15:42:03.775 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-23T15:42:03.811 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-23T15:42:04.126 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-23T15:42:04.158 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-23T15:42:04.248 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-23T15:42:04.279 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 126/220 2024-03-23T15:42:04.316 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-23T15:42:04.782 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-23T15:42:04.811 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-23T15:42:04.882 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-23T15:42:04.904 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/220 2024-03-23T15:42:07.150 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-23T15:42:07.185 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-23T15:42:07.723 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-23T15:42:07.755 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-23T15:42:07.850 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-23T15:42:07.850 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T15:42:07.851 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T15:42:07.851 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:07.890 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-23T15:42:13.479 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-23T15:42:13.479 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-03-23T15:42:13.479 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-03-23T15:42:13.479 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-03-23T15:42:13.479 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-03-23T15:42:13.479 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-03-23T15:42:13.480 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-03-23T15:42:13.480 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-03-23T15:42:13.480 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:13.641 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-23T15:42:13.660 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-23T15:42:13.660 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:13.660 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T15:42:13.660 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:15.152 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-23T15:42:15.177 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-23T15:42:15.178 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:15.178 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T15:42:15.178 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T15:42:15.178 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T15:42:15.178 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:15.403 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-23T15:42:15.425 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-23T15:42:15.425 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:15.425 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T15:42:15.425 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T15:42:15.426 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T15:42:15.426 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:15.862 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-23T15:42:15.885 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-23T15:42:15.885 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:15.885 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T15:42:15.885 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T15:42:15.885 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T15:42:15.885 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:15.981 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-23T15:42:16.448 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-23T15:42:16.471 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-23T15:42:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 140/220 2024-03-23T15:42:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 141/220 2024-03-23T15:42:16.586 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-23T15:42:16.818 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-23T15:42:16.845 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-23T15:42:17.805 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-23T15:42:17.829 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-23T15:42:17.930 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-23T15:42:18.092 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 145/220 2024-03-23T15:42:18.114 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-23T15:42:18.136 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-23T15:42:18.137 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:18.137 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T15:42:18.137 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T15:42:18.137 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T15:42:18.137 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:18.173 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-23T15:42:18.184 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-23T15:42:18.227 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 148/220 2024-03-23T15:42:19.081 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-23T15:42:19.108 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-23T15:42:19.131 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-23T15:42:19.131 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:19.132 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T15:42:19.132 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T15:42:19.132 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T15:42:19.132 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:19.176 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-23T15:42:19.197 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-23T15:42:19.197 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:19.197 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T15:42:19.197 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:19.251 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-23T15:42:19.273 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-23T15:42:19.273 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:19.273 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T15:42:19.273 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T15:42:19.273 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T15:42:19.274 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:19.577 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-23T15:42:19.599 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-23T15:42:19.599 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:19.600 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T15:42:19.600 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T15:42:19.600 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T15:42:19.600 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:42:20.324 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-23T15:42:20.324 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-03-23T15:42:20.324 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-03-23T15:42:20.324 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-03-23T15:42:20.324 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-03-23T15:42:20.324 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-03-23T15:42:20.325 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-03-23T15:42:20.325 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-03-23T15:42:20.325 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:20.490 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-23T15:42:20.505 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-23T15:42:20.505 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:20.505 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T15:42:20.506 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:21.993 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-23T15:42:22.018 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-23T15:42:22.018 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:22.018 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T15:42:22.019 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T15:42:22.019 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T15:42:22.019 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:22.252 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-23T15:42:22.274 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-23T15:42:22.274 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:22.274 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T15:42:22.274 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T15:42:22.275 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T15:42:22.275 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:22.711 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-23T15:42:22.733 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-23T15:42:22.734 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:22.734 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T15:42:22.734 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T15:42:22.734 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T15:42:22.734 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:22.821 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-23T15:42:23.231 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-23T15:42:23.262 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-23T15:42:23.293 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 140/220 2024-03-23T15:42:23.355 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 141/220 2024-03-23T15:42:23.385 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-23T15:42:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-23T15:42:23.653 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-23T15:42:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/220 2024-03-23T15:42:24.073 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/220 2024-03-23T15:42:24.103 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 156/220 2024-03-23T15:42:24.130 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-23T15:42:24.312 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 158/220 2024-03-23T15:42:24.429 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-23T15:42:24.479 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 160/220 2024-03-23T15:42:24.539 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-23T15:42:24.587 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-23T15:42:24.615 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-23T15:42:24.622 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-23T15:42:24.645 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-23T15:42:24.654 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 164/220 2024-03-23T15:42:24.695 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-23T15:42:24.724 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 166/220 2024-03-23T15:42:24.745 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-23T15:42:24.790 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 167/220 2024-03-23T15:42:24.858 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-23T15:42:24.891 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-23T15:42:24.908 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 145/220 2024-03-23T15:42:24.939 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-23T15:42:24.962 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-23T15:42:24.962 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:24.962 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T15:42:24.962 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T15:42:24.962 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T15:42:24.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:25.006 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-23T15:42:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-23T15:42:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 148/220 2024-03-23T15:42:25.507 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-23T15:42:25.548 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-23T15:42:25.567 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-23T15:42:25.935 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-23T15:42:25.967 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-23T15:42:25.989 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-23T15:42:25.989 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:25.989 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T15:42:25.989 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T15:42:25.989 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T15:42:25.989 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:26.034 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-23T15:42:26.056 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-23T15:42:26.056 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:26.056 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T15:42:26.056 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:26.110 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-23T15:42:26.132 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-23T15:42:26.132 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:26.132 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T15:42:26.133 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T15:42:26.133 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T15:42:26.133 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:26.444 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-23T15:42:26.466 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-23T15:42:26.466 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T15:42:26.466 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T15:42:26.467 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T15:42:26.467 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T15:42:26.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:42:30.891 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/220 2024-03-23T15:42:30.932 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/220 2024-03-23T15:42:30.962 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 156/220 2024-03-23T15:42:30.990 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-23T15:42:31.169 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 158/220 2024-03-23T15:42:31.281 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-23T15:42:31.322 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 160/220 2024-03-23T15:42:31.378 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-23T15:42:31.438 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-23T15:42:31.473 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-23T15:42:31.506 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 164/220 2024-03-23T15:42:31.547 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-23T15:42:31.577 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 166/220 2024-03-23T15:42:31.646 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 167/220 2024-03-23T15:42:31.704 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-23T15:42:31.742 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-23T15:42:32.313 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-23T15:42:32.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-23T15:42:32.368 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-23T15:42:32.629 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 171/220 2024-03-23T15:42:39.449 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 171/220 2024-03-23T15:43:14.293 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 172/220 2024-03-23T15:43:14.917 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 173/220 2024-03-23T15:43:15.094 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 174/220 2024-03-23T15:43:19.902 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 175/220 2024-03-23T15:43:21.062 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 172/220 2024-03-23T15:43:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 173/220 2024-03-23T15:43:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 174/220 2024-03-23T15:43:22.225 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 176/220 2024-03-23T15:43:26.669 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 175/220 2024-03-23T15:43:28.725 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 176/220 2024-03-23T15:43:31.810 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 177/220 2024-03-23T15:43:38.359 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 177/220 2024-03-23T15:43:59.444 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 178/220 2024-03-23T15:44:06.104 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 178/220 2024-03-23T15:44:16.467 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 179/220 2024-03-23T15:44:23.102 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 179/220 2024-03-23T15:45:23.575 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 180/220 2024-03-23T15:45:24.139 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 181/220 2024-03-23T15:45:24.187 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 182/220 2024-03-23T15:45:28.909 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 183/220 2024-03-23T15:45:30.330 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 180/220 2024-03-23T15:45:31.168 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 181/220 2024-03-23T15:45:31.468 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 182/220 2024-03-23T15:45:34.731 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 184/220 2024-03-23T15:45:35.085 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 185/220 2024-03-23T15:45:35.436 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 186/220 2024-03-23T15:45:35.837 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 187/220 2024-03-23T15:45:36.092 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 188/220 2024-03-23T15:45:36.172 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 189/220 2024-03-23T15:45:36.498 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 183/220 2024-03-23T15:45:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 190/220 2024-03-23T15:45:42.095 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 191/220 2024-03-23T15:45:42.194 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 184/220 2024-03-23T15:45:42.279 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 185/220 2024-03-23T15:45:42.377 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 186/220 2024-03-23T15:45:42.513 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 187/220 2024-03-23T15:45:42.571 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 188/220 2024-03-23T15:45:42.652 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 189/220 2024-03-23T15:45:43.179 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 192/220 2024-03-23T15:45:43.282 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-23T15:45:43.330 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-23T15:45:43.428 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-23T15:45:43.461 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-23T15:45:43.486 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-23T15:45:43.511 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-23T15:45:43.537 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-23T15:45:43.538 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-23T15:45:43.609 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-23T15:45:43.647 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-23T15:45:43.647 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-23T15:45:43.723 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-23T15:45:43.760 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-23T15:45:43.791 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-23T15:45:43.816 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-23T15:45:43.841 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-23T15:45:43.875 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-23T15:45:43.930 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-23T15:45:43.962 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-23T15:45:43.988 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-23T15:45:44.013 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-23T15:45:44.038 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-23T15:45:44.063 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-23T15:45:44.069 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-23T15:45:44.069 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-23T15:45:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-23T15:45:44.250 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-23T15:45:44.333 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-23T15:45:44.357 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-23T15:45:44.389 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-23T15:45:44.417 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-23T15:45:44.442 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-23T15:45:44.508 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-23T15:45:48.197 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 190/220 2024-03-23T15:45:48.340 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 191/220 2024-03-23T15:45:49.188 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 192/220 2024-03-23T15:45:49.280 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-23T15:45:49.324 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-23T15:45:49.428 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-23T15:45:49.459 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-23T15:45:49.484 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-23T15:45:49.510 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-23T15:45:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-23T15:45:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-23T15:45:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-23T15:45:49.671 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-23T15:45:49.671 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-23T15:45:49.747 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-23T15:45:49.779 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-23T15:45:49.807 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-23T15:45:49.832 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-23T15:45:49.865 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-23T15:45:49.915 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-23T15:45:49.963 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-23T15:45:49.996 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-23T15:45:50.021 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-23T15:45:50.046 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-23T15:45:50.080 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-23T15:45:50.105 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-23T15:45:50.111 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-23T15:45:50.112 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-23T15:45:50.292 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-23T15:45:50.292 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-23T15:45:50.383 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-23T15:45:50.408 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-23T15:45:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-23T15:45:50.467 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-23T15:45:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-23T15:45:50.568 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 3/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 5/220 2024-03-23T15:45:53.925 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 6/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 8/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 9/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 10/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 12/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 14/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 15/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 16/220 2024-03-23T15:45:53.926 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 18/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 19/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 20/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 21/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 22/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 23/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 26/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/220 2024-03-23T15:45:53.927 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 28/220 2024-03-23T15:45:53.928 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/220 2024-03-23T15:45:53.928 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 30/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 31/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 32/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 33/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 34/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 35/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 36/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 37/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 38/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 39/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 40/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 41/220 2024-03-23T15:45:53.948 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 42/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 43/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 44/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 45/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 46/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 47/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 48/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 49/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 50/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 51/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 52/220 2024-03-23T15:45:53.949 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 53/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 54/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 55/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 56/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 57/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 58/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 59/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 60/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 61/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-23T15:45:53.950 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-23T15:45:53.951 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-23T15:45:53.952 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-23T15:45:53.953 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-23T15:45:53.954 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-23T15:45:53.955 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-23T15:45:53.956 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 138/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-23T15:45:53.957 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 146/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-23T15:45:53.958 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-23T15:45:53.959 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 173/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 175/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 179/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 181/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 183/220 2024-03-23T15:45:53.960 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 185/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-23T15:45:53.961 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-23T15:45:53.962 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-23T15:45:53.963 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-23T15:45:56.930 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-23T15:45:56.930 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.931 INFO:teuthology.orchestra.run.smithi157.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T15:45:56.932 INFO:teuthology.orchestra.run.smithi157.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.933 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.934 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.935 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-16.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T15:45:56.936 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.937 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T15:45:56.938 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.939 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T15:45:56.940 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T15:45:56.941 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T15:45:56.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T15:45:56.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T15:45:56.944 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T15:45:56.945 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T15:45:56.946 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T15:45:56.946 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:45:56.946 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-23T15:45:57.610 DEBUG:teuthology.parallel:result is None 2024-03-23T15:46:00.171 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-23T15:46:00.171 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/220 2024-03-23T15:46:00.171 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 3/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 5/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 6/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 8/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 9/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 10/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 12/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 14/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 15/220 2024-03-23T15:46:00.172 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 16/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 18/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 19/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 20/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 21/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 22/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 23/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 26/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 28/220 2024-03-23T15:46:00.173 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/220 2024-03-23T15:46:00.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 30/220 2024-03-23T15:46:00.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 31/220 2024-03-23T15:46:00.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 32/220 2024-03-23T15:46:00.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 33/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 34/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 35/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 36/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 37/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 38/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 39/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 40/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 41/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 42/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 43/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 44/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 45/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 46/220 2024-03-23T15:46:00.194 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 47/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 48/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 49/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 50/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 51/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 52/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 53/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 54/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 55/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 56/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 57/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 58/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 59/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 60/220 2024-03-23T15:46:00.195 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 61/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-23T15:46:00.196 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-23T15:46:00.197 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-23T15:46:00.198 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-23T15:46:00.199 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-23T15:46:00.200 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 138/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-23T15:46:00.201 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 146/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-23T15:46:00.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-23T15:46:00.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 173/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 175/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 179/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 181/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 183/220 2024-03-23T15:46:00.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 185/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-23T15:46:00.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-23T15:46:00.206 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-23T15:46:00.207 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-23T15:46:00.207 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-23T15:46:00.207 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-23T15:46:00.207 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-23T15:46:00.207 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-23T15:46:03.162 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-23T15:46:03.162 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:03.162 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-03-23T15:46:03.162 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T15:46:03.162 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.163 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T15:46:03.164 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.165 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.166 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T15:46:03.167 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: jq-1.6-16.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T15:46:03.168 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T15:46:03.169 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T15:46:03.170 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T15:46:03.171 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T15:46:03.172 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T15:46:03.173 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.174 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T15:46:03.175 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T15:46:03.176 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T15:46:03.177 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T15:46:03.177 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:03.177 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-23T15:46:03.889 DEBUG:teuthology.parallel:result is None 2024-03-23T15:46:03.890 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T15:46:04.111 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T15:46:04.133 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-23T15:46:04.133 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-23T15:46:04.133 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-23T15:46:04.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T15:46:04.273 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T15:46:04.299 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-23T15:46:04.299 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-23T15:46:04.299 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-23T15:46:04.301 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T15:46:04.301 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:04.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T15:46:04.331 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:04.331 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T15:46:04.372 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T15:46:04.373 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:04.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T15:46:04.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T15:46:04.467 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:04.467 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T15:46:04.496 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T15:46:04.565 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T15:46:04.565 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:04.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T15:46:04.592 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T15:46:04.659 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:04.659 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T15:46:04.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T15:46:04.756 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T15:46:04.756 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:04.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T15:46:04.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T15:46:04.850 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:04.850 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T15:46:04.878 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T15:46:04.947 INFO:teuthology.run_tasks:Running task ceph... 2024-03-23T15:46:05.033 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-23T15:46:05.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-03-23T15:46:05.035 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-03-23T15:46:05.063 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-23T15:46:05.063 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T15:46:05.102 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T15:46:05.129 INFO:tasks.ceph:Creating extra log directories... 2024-03-23T15:46:05.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T15:46:05.169 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T15:46:05.202 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-23T15:46:05.202 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-23T15:46:05.202 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-23T15:46:05.202 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:05.202 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T15:46:05.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T15:46:05.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T15:46:05.391 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:05.392 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T15:46:05.406 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T15:46:05.522 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T15:46:05.554 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-23T15:46:05.554 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-23T15:46:05.555 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460', 'branch': 'wip-batrick-testing-20240323.003144-squid', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7618460', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240323.003144-squid'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240323.003144-squid', 'sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 4284, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'sleep_before_teardown': 0, 'subset': '405/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240323.003144-squid', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph.git', 'suite_sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARsO9mkf9Sa31l8h16xrU7JaFkB1i1hbbhND5Jh393VvdrfJQgScMrL5q2WDTpBNPcCZECoHnbd2bnmqxUjiXQ=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+9px7MnSxpoSkDwUcXr7nRW8kSItohx68roYhA2tWLsD7fcV7x+aE719oUsVNHglGXaCzm4ON+32TRNsGUUKg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['suites/fsstress.sh']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-23_09:56:53', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-23T15:46:05.555 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T15:46:05.557 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T15:46:05.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T15:46:05.612 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T15:46:05.672 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:05.672 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T15:46:05.693 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T15:46:05.693 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-03-23T15:46:05.748 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T15:46:05.748 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:05.748 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-23T15:46:05.748 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:05.748 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:05.748 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-23 15:45:50.889792728 +0000 2024-03-23T15:46:05.749 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-23 15:37:05.709718031 +0000 2024-03-23T15:46:05.749 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-23 15:37:05.709718031 +0000 2024-03-23T15:46:05.749 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-23 15:37:05.709718031 +0000 2024-03-23T15:46:05.749 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T15:46:05.813 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-23T15:46:05.813 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-23T15:46:05.813 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000204429 s, 2.5 MB/s 2024-03-23T15:46:05.815 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T15:46:05.871 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-23 15:45:50.890792707 +0000 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-23 15:37:06.070710414 +0000 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-23 15:37:06.070710414 +0000 2024-03-23T15:46:05.926 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-23 15:37:06.070710414 +0000 2024-03-23T15:46:05.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T15:46:05.991 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-23T15:46:05.991 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-23T15:46:05.992 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000177956 s, 2.9 MB/s 2024-03-23T15:46:05.993 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T15:46:06.048 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-03-23T15:46:06.104 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T15:46:06.465 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:06.465 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-23T15:46:06.465 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:06.466 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:06.466 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-23 15:45:50.890792707 +0000 2024-03-23T15:46:06.466 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-23 15:37:06.432702776 +0000 2024-03-23T15:46:06.466 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-23 15:37:06.432702776 +0000 2024-03-23T15:46:06.466 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-23 15:37:06.432702776 +0000 2024-03-23T15:46:06.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T15:46:06.493 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-23T15:46:06.493 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-23T15:46:06.493 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000184059 s, 2.8 MB/s 2024-03-23T15:46:06.494 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T15:46:06.551 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-23 15:45:50.891792685 +0000 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-23 15:37:06.765695750 +0000 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-23 15:37:06.765695750 +0000 2024-03-23T15:46:06.607 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-23 15:37:06.765695750 +0000 2024-03-23T15:46:06.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T15:46:06.673 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-23T15:46:06.674 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-23T15:46:06.674 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000176098 s, 2.9 MB/s 2024-03-23T15:46:06.675 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T15:46:06.731 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:06.731 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:06.731 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T15:46:06.752 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T15:46:06.753 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-03-23T15:46:06.810 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-23 15:45:44.870901360 +0000 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-23 15:37:05.095131497 +0000 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-23 15:37:05.095131497 +0000 2024-03-23T15:46:06.811 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-23 15:37:05.095131497 +0000 2024-03-23T15:46:06.812 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T15:46:06.879 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-23T15:46:06.880 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-23T15:46:06.880 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000114165 s, 4.5 MB/s 2024-03-23T15:46:06.881 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T15:46:06.938 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-03-23T15:46:06.995 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T15:46:06.995 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:06.995 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-23T15:46:06.995 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:06.996 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:06.996 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-23 15:45:44.870901360 +0000 2024-03-23T15:46:06.996 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-23 15:37:05.422125513 +0000 2024-03-23T15:46:06.996 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-23 15:37:05.422125513 +0000 2024-03-23T15:46:06.996 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-23 15:37:05.422125513 +0000 2024-03-23T15:46:06.996 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T15:46:07.062 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-23T15:46:07.062 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-23T15:46:07.062 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000108989 s, 4.7 MB/s 2024-03-23T15:46:07.064 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T15:46:07.120 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-03-23T15:46:07.177 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T15:46:07.177 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-23 15:45:44.870901360 +0000 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-23 15:37:05.759119347 +0000 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-23 15:37:05.759119347 +0000 2024-03-23T15:46:07.178 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-23 15:37:05.759119347 +0000 2024-03-23T15:46:07.178 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T15:46:07.245 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-23T15:46:07.245 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-23T15:46:07.245 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.8774e-05 s, 5.2 MB/s 2024-03-23T15:46:07.246 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T15:46:07.302 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-23 15:45:44.871901341 +0000 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-23 15:37:06.089113309 +0000 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-23 15:37:06.089113309 +0000 2024-03-23T15:46:07.358 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-23 15:37:06.089113309 +0000 2024-03-23T15:46:07.359 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T15:46:07.424 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-23T15:46:07.424 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-23T15:46:07.424 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 7.6963e-05 s, 6.7 MB/s 2024-03-23T15:46:07.425 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T15:46:07.481 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:07.481 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T15:46:07.481 INFO:tasks.ceph:Generating config... 2024-03-23T15:46:07.482 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-23T15:46:07.482 INFO:tasks.ceph:[client] debug client = 20 2024-03-23T15:46:07.482 INFO:tasks.ceph:[client] debug ms = 1 2024-03-23T15:46:07.482 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-23T15:46:07.483 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-23T15:46:07.483 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-03-23T15:46:07.483 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-03-23T15:46:07.483 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-23T15:46:07.483 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-23T15:46:07.484 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-23T15:46:07.484 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-23T15:46:07.484 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-23T15:46:07.484 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-23T15:46:07.484 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-23T15:46:07.484 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-23T15:46:07.485 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-23T15:46:07.485 INFO:tasks.ceph:Setting up mon.a... 2024-03-23T15:46:07.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-23T15:46:07.539 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2024-03-23T15:46:07.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-23T15:46:07.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T15:46:07.616 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129'), ('mon.b', '172.21.15.157'), ('mon.c', '[v2:172.21.15.157:3301,v1:172.21.15.157:6790]')] 2024-03-23T15:46:07.616 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-23T15:46:07.617 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:07.617 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T15:46:07.672 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --add b 172.21.15.157 --addv c '[v2:172.21.15.157:3301,v1:172.21.15.157:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid 24ba5e35-b5b5-4c1d-a218-52a63f3af0eb 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:fsid 24ba5e35-b5b5-4c1d-a218-52a63f3af0eb 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2024-03-23T15:46:07.756708+0000 2024-03-23T15:46:07.758 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-03-23T15:46:07.756708+0000 2024-03-23T15:46:07.759 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 17 (quincy) 2024-03-23T15:46:07.759 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 1 2024-03-23T15:46:07.759 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-03-23T15:46:07.759 INFO:teuthology.orchestra.run.smithi129.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2024-03-23T15:46:07.759 INFO:teuthology.orchestra.run.smithi129.stdout:2: [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] mon.c 2024-03-23T15:46:07.759 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-23T15:46:07.760 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T15:46:07.775 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 24ba5e35-b5b5-4c1d-a218-52a63f3af0eb... 2024-03-23T15:46:07.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T15:46:07.818 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T15:46:07.865 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-03-23T15:46:07.865 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.866 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-03-23T15:46:07.867 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]" 2024-03-23T15:46:07.868 INFO:teuthology.orchestra.run.smithi129.stdout: mds beacon grace = 60 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: mds heartbeat grace = 60 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd heartbeat grace = 60 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 24ba5e35-b5b5-4c1d-a218-52a63f3af0eb 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression mode = aggressive 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 20 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 20 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 10 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T15:46:07.871 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: osd fast shutdown = False 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: osd op complaint time = 180 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-03-23T15:46:07.872 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: debug client = 20 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T15:46:07.873 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon op complaint time = 120 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd crush smoke test = False 2024-03-23T15:46:07.874 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: client mount timeout = 600 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: debug client = 20 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: rados mon op timeout = 900 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout: rados osd op timeout = 900 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-03-23T15:46:07.875 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.e] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.f] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds = 20 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds balancer = 20 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: mds debug frag = True 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: mds debug scatterstat = True 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: mds op complaint time = 180 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: mds valgrind exit = True 2024-03-23T15:46:07.876 INFO:teuthology.orchestra.run.smithi129.stdout: mds verify scatter = True 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi129.stdout: osd op complaint time = 180 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi129.stdout: rados mon op timeout = 900 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi129.stdout: rados osd op timeout = 900 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.877 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T15:46:07.878 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.879 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.129,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]" 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: mds beacon grace = 60 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: mds heartbeat grace = 60 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: osd heartbeat grace = 60 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 24ba5e35-b5b5-4c1d-a218-52a63f3af0eb 2024-03-23T15:46:07.880 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-03-23T15:46:07.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-03-23T15:46:07.882 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression mode = aggressive 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: osd fast shutdown = False 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-03-23T15:46:07.883 INFO:teuthology.orchestra.run.smithi157.stdout: osd op complaint time = 180 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug client = 20 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-23T15:46:07.884 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T15:46:07.885 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: mon down mkfs grace = 300 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: mon op complaint time = 120 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd crush smoke test = False 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T15:46:07.886 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout: client mount timeout = 600 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout: debug client = 20 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout: rados mon op timeout = 900 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout: rados osd op timeout = 900 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.a] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.c] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.e] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.b] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.d] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.f] 2024-03-23T15:46:07.887 INFO:teuthology.orchestra.run.smithi157.stdout:[mds] 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: debug mds = 20 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: debug mds balancer = 20 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: mds debug frag = True 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: mds debug scatterstat = True 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: mds op complaint time = 180 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: mds valgrind exit = True 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: mds verify scatter = True 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: osd op complaint time = 180 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: rados mon op timeout = 900 2024-03-23T15:46:07.888 INFO:teuthology.orchestra.run.smithi157.stdout: rados osd op timeout = 900 2024-03-23T15:46:07.889 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-23T15:46:07.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-23T15:46:07.985 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-23T15:46:07.985 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:07.985 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-23T15:46:08.000 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:08.000 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-23T15:46:08.055 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T15:46:08.055 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:08.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T15:46:08.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T15:46:08.134 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:08.134 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:08.189 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T15:46:08.189 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:08.189 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T15:46:08.189 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T15:46:08.231 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:08.231 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:08.285 INFO:tasks.ceph:Setting up mon nodes... 2024-03-23T15:46:08.286 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-23T15:46:08.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T15:46:08.341 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T15:46:08.343 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T15:46:08.403 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T15:46:08.405 INFO:tasks.ceph:Setting up mds nodes... 2024-03-23T15:46:08.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T15:46:08.461 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T15:46:08.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-23T15:46:08.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T15:46:08.583 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T15:46:08.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-23T15:46:08.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-23T15:46:08.709 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-23T15:46:08.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-23T15:46:08.738 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T15:46:08.795 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T15:46:08.797 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-23T15:46:08.824 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T15:46:08.918 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T15:46:08.921 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-23T15:46:08.947 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-23T15:46:09.042 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-23T15:46:09.045 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-23T15:46:09.072 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-23T15:46:09.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-23T15:46:09.115 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-23T15:46:09.130 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-23T15:46:09.131 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T15:46:09.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-23T15:46:09.197 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:09.197 INFO:tasks.ceph:role: osd.0 2024-03-23T15:46:09.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T15:46:09.197 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T15:46:09.271 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:09.272 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:09.781 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-23T15:46:09.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-23T15:46:09.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-23T15:46:09.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-23T15:46:09.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-23T15:46:10.029 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:10.029 INFO:tasks.ceph:role: osd.1 2024-03-23T15:46:10.030 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T15:46:10.030 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T15:46:10.112 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:10.112 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:10.113 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:10.113 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:10.113 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:10.113 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:10.557 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-23T15:46:10.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-23T15:46:10.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-23T15:46:10.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-23T15:46:10.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-23T15:46:10.746 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:10.746 INFO:tasks.ceph:role: osd.2 2024-03-23T15:46:10.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T15:46:10.746 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T15:46:10.820 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:10.821 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:11.310 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-23T15:46:11.311 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-23T15:46:11.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-23T15:46:11.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-23T15:46:11.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-23T15:46:11.542 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:11.542 INFO:tasks.ceph:role: osd.3 2024-03-23T15:46:11.543 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T15:46:11.543 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T15:46:11.621 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:11.622 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:12.129 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-23T15:46:12.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-23T15:46:12.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-23T15:46:12.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-23T15:46:12.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:12.423+0000 7f8516d47640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-23T15:46:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:12.423+0000 7f8516d47640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-23T15:46:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:12.423+0000 7f8516d47640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T15:46:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:12.423+0000 7f8516d47640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T15:46:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:12.423+0000 7f8516d47640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T15:46:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:12.424+0000 7f8516d47640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-23T15:46:15.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-23T15:46:15.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:15.398+0000 7f876ca13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-23T15:46:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:15.398+0000 7f876ca13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-23T15:46:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:15.398+0000 7f876ca13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T15:46:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:15.398+0000 7f876ca13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T15:46:15.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:15.398+0000 7f876ca13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T15:46:15.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:15.399+0000 7f876ca13640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-23T15:46:18.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-23T15:46:18.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:18.374+0000 7f5b17b93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-23T15:46:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:18.375+0000 7f5b17b93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-23T15:46:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:18.375+0000 7f5b17b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T15:46:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:18.375+0000 7f5b17b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T15:46:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:18.375+0000 7f5b17b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T15:46:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:18.375+0000 7f5b17b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-23T15:46:21.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-23T15:46:21.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:21.346+0000 7f819bd28640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-23T15:46:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:21.346+0000 7f819bd28640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-23T15:46:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:21.346+0000 7f819bd28640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T15:46:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:21.346+0000 7f819bd28640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T15:46:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:21.346+0000 7f819bd28640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T15:46:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:21.347+0000 7f819bd28640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-23T15:46:24.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-23T15:46:24.228 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-23T15:46:24.260 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:24.261 INFO:tasks.ceph:role: osd.4 2024-03-23T15:46:24.261 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T15:46:24.261 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T15:46:24.331 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:24.331 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:24.331 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:24.331 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:24.331 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:24.331 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:24.332 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:24.332 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:24.332 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:24.332 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:24.366 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-23T15:46:24.368 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-23T15:46:24.368 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-23T15:46:24.446 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-23T15:46:24.519 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-23T15:46:24.587 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:24.588 INFO:tasks.ceph:role: osd.5 2024-03-23T15:46:24.588 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T15:46:24.588 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:24.656 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:24.657 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:24.688 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-23T15:46:24.690 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-23T15:46:24.690 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-23T15:46:24.766 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-23T15:46:24.837 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-23T15:46:24.906 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:24.906 INFO:tasks.ceph:role: osd.6 2024-03-23T15:46:24.906 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T15:46:24.907 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T15:46:24.973 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:24.973 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:24.973 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:24.973 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:24.973 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:24.974 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:24.974 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:24.974 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:24.974 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:24.974 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:25.006 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-23T15:46:25.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-23T15:46:25.007 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-23T15:46:25.078 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-23T15:46:25.150 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-23T15:46:25.217 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T15:46:25.217 INFO:tasks.ceph:role: osd.7 2024-03-23T15:46:25.217 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T15:46:25.217 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T15:46:25.283 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T15:46:25.284 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T15:46:25.284 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T15:46:25.284 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T15:46:25.285 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T15:46:25.285 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-23T15:46:25.285 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T15:46:25.285 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T15:46:25.285 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T15:46:25.285 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T15:46:25.316 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-23T15:46:25.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-23T15:46:25.318 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-23T15:46:25.388 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-23T15:46:25.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:25.559 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:25.556+0000 7f5791010640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-23T15:46:25.559 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:25.556+0000 7f5791010640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-23T15:46:25.559 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:25.556+0000 7f5791010640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T15:46:25.559 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:25.556+0000 7f5791010640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T15:46:25.559 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:25.556+0000 7f5791010640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T15:46:25.559 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:25.556+0000 7f5791010640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-23T15:46:28.399 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-23T15:46:28.431 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:28.522 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:28.519+0000 7fb2d6819640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-23T15:46:28.522 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:28.520+0000 7fb2d6819640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-23T15:46:28.522 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:28.520+0000 7fb2d6819640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T15:46:28.522 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:28.520+0000 7fb2d6819640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T15:46:28.522 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:28.520+0000 7fb2d6819640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T15:46:28.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:28.520+0000 7fb2d6819640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-23T15:46:31.369 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-23T15:46:31.401 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:31.495 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:31.492+0000 7f0eda8d1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-23T15:46:31.495 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:31.492+0000 7f0eda8d1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-23T15:46:31.496 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:31.492+0000 7f0eda8d1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T15:46:31.496 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:31.493+0000 7f0eda8d1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T15:46:31.496 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:31.493+0000 7f0eda8d1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T15:46:31.496 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:31.493+0000 7f0eda8d1640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-23T15:46:34.339 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-23T15:46:34.372 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:34.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:34.462+0000 7ff008e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-23T15:46:34.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:34.462+0000 7ff008e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-23T15:46:34.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:34.462+0000 7ff008e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T15:46:34.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:34.462+0000 7ff008e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T15:46:34.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:34.462+0000 7ff008e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T15:46:34.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:34.462+0000 7ff008e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-23T15:46:35.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:46:35.560 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:46:37.314 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-23T15:46:37.344 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-23T15:46:37.345 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-23T15:46:37.377 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-23T15:46:37.447 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-23T15:46:37.518 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-23T15:46:37.587 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-23T15:46:37.656 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-23T15:46:37.724 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-23T15:46:37.793 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:37.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-23T15:46:37.860 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:37.860 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-23T15:46:37.892 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:37.892 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-23T15:46:37.963 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:37.963 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-23T15:46:38.033 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:38.033 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-23T15:46:38.102 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:38.104 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-23T15:46:38.171 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:38.171 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-23T15:46:38.239 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:38.240 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-23T15:46:38.308 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:38.308 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-23T15:46:38.376 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:38.376 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-23T15:46:38.393 INFO:tasks.ceph:Adding keys to all mons... 2024-03-23T15:46:38.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T15:46:38.436 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T15:46:38.461 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.y] 2024-03-23T15:46:38.461 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5l52E9FBAAOeq2/V4Rsf4+SUs3b8fc8A== 2024-03-23T15:46:38.461 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-03-23T15:46:38.461 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5lG11iGxAAWckOgIwu2Jw+S08zzdddsA== 2024-03-23T15:46:38.461 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-03-23T15:46:38.461 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5lA5e0IhAATCZLd2Y0D3iBxRSC1rBq8w== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.e] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5lFmwoKhAAkYYNI+KB9Tb/gqyVWtWF0g== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBE+f5lc35OGRAAcsv4lcYq8FhZ+S7R4+LU1A== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBH+f5lLxLSFxAAqEwbE3c5wh8apiEd2cnywg== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBK+f5lLLVoFhAANvFocD5Hxn4jc0egSMxfBQ== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBN+f5lgvu0FBAA7/NzWsXWUf533/tdJhaMZg== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5lNv31FxAAD4oMlf6evxld4Z4AM008qQ== 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-03-23T15:46:38.462 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5lSctFLxAAEhvuasDxntGiH1uNY04Mjg== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBA+f5lT/OrNhAAonV4oAxbDiVZWAyUnJdBYA== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.f] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBB+f5lNItzAhAAIv85KK69oZw1mAg1UaPH1A== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBR+f5lUTgyIRAADIeaowS7w4ZhDeYzq7KmzA== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBU+f5lP0oDHxAAjvHGd5jnPcUDfhz2jPdfwg== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.6] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBX+f5lGmpnHRAA5QIgI4czU6L1s6Abd1nTIA== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.7] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBa+f5lRuWUGxAAbbutb44XXvh1lgNAhQ0C7g== 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-03-23T15:46:38.463 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBB+f5lbzDOBhAAdYKDsvaQjUMg8ui+YY3dCg== 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5l52E9FBAAOeq2/V4Rsf4+SUs3b8fc8A== 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.a] 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5lG11iGxAAWckOgIwu2Jw+S08zzdddsA== 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.c] 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5lA5e0IhAATCZLd2Y0D3iBxRSC1rBq8w== 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.e] 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5lFmwoKhAAkYYNI+KB9Tb/gqyVWtWF0g== 2024-03-23T15:46:38.465 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBE+f5lc35OGRAAcsv4lcYq8FhZ+S7R4+LU1A== 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBH+f5lLxLSFxAAqEwbE3c5wh8apiEd2cnywg== 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBK+f5lLLVoFhAANvFocD5Hxn4jc0egSMxfBQ== 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBN+f5lgvu0FBAA7/NzWsXWUf533/tdJhaMZg== 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2024-03-23T15:46:38.466 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5lNv31FxAAD4oMlf6evxld4Z4AM008qQ== 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.b] 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5lSctFLxAAEhvuasDxntGiH1uNY04Mjg== 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.d] 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBA+f5lT/OrNhAAonV4oAxbDiVZWAyUnJdBYA== 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.f] 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBB+f5lNItzAhAAIv85KK69oZw1mAg1UaPH1A== 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBR+f5lUTgyIRAADIeaowS7w4ZhDeYzq7KmzA== 2024-03-23T15:46:38.467 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBU+f5lP0oDHxAAjvHGd5jnPcUDfhz2jPdfwg== 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBX+f5lGmpnHRAA5QIgI4czU6L1s6Abd1nTIA== 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBa+f5lRuWUGxAAbbutb44XXvh1lgNAhQ0C7g== 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2024-03-23T15:46:38.468 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBB+f5lbzDOBhAAdYKDsvaQjUMg8ui+YY3dCg== 2024-03-23T15:46:38.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T15:46:38.508 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T15:46:38.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:38.564 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:38.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:38.621 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:38.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:38.677 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:38.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.734 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.787 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.796 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.861 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.916 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:38.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T15:46:38.978 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T15:46:39.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:39.040 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:39.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:39.100 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:39.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:39.158 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T15:46:39.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.216 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.271 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.326 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T15:46:39.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T15:46:39.435 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T15:46:39.490 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-23T15:46:39.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-23T15:46:39.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T15:46:39.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-23T15:46:39.644 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-23T15:46:39.671 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T15:46:39.767 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-23T15:46:39.794 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-23T15:46:39.862 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T15:46:39.959 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-23T15:46:39.986 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:39.989 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T15:46:40.040 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-23T15:46:40.041 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-23T15:46:40.041 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-23T15:46:40.041 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-23T15:46:40.044 INFO:tasks.ceph.mon.a:Started 2024-03-23T15:46:40.044 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-23T15:46:40.044 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-23T15:46:40.044 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-23T15:46:40.083 INFO:tasks.ceph.mon.b:Started 2024-03-23T15:46:40.084 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-23T15:46:40.084 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-23T15:46:40.084 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-23T15:46:40.086 INFO:tasks.ceph.mon.c:Started 2024-03-23T15:46:40.086 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-23T15:46:40.086 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-23T15:46:40.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-23T15:46:40.088 INFO:tasks.ceph.mgr.y:Started 2024-03-23T15:46:40.089 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-23T15:46:40.089 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-23T15:46:40.091 INFO:tasks.ceph.mgr.x:Started 2024-03-23T15:46:40.091 INFO:tasks.ceph:Setting crush tunables to default 2024-03-23T15:46:40.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2024-03-23T15:46:40.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.189+0000 7f6f04815640 1 Processor -- start 2024-03-23T15:46:40.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.189+0000 7f6f04815640 1 -- start start 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.190+0000 7f6f04815640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 0x7f6f0006c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.190+0000 7f6f04815640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 0x7f6f000cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.190+0000 7f6f04815640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f00154c90 con 0x7f6f0015d4f0 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.190+0000 7f6f04815640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6f0015b940 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.190+0000 7f6f04815640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0006bb90 con 0x7f6f0006d570 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.191+0000 7f6ef7fff640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 msgr2=0x7f6f0006c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.191+0000 7f6ef7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 0x7f6f0006c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.191+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 msgr2=0x7f6f000cc290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.191+0000 7f6efe59c640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f6f0015d4f0 legacy=0x7f6f0015a4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-23T15:46:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.191+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 0x7f6f000cc290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:40.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.392+0000 7f6ef7fff640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 msgr2=0x7f6f0006c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2024-03-23T15:46:40.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.392+0000 7f6ef7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 0x7f6f0006c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:40.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.392+0000 7f6efe59c640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f6f0015d4f0 legacy=0x7f6f0015a4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-23T15:46:40.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.392+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 msgr2=0x7f6f000cc290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:40.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.392+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 0x7f6f000cc290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:40.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.793+0000 7f6ef7fff640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 msgr2=0x7f6f0006c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2024-03-23T15:46:40.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.793+0000 7f6ef7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 0x7f6f0006c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:40.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.793+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 msgr2=0x7f6f000cc290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:40.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.793+0000 7f6efe59c640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f6f0015d4f0 legacy=0x7f6f0015a4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-23T15:46:40.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:40.793+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 0x7f6f000cc290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:41.594+0000 7f6ef7fff640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 msgr2=0x7f6f0006c5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.129:3300/0 2024-03-23T15:46:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:41.594+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 msgr2=0x7f6f000cc290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:41.594+0000 7f6efe59c640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f6f0015d4f0 legacy=0x7f6f0015a4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.157:6789/0 2024-03-23T15:46:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:41.594+0000 7f6ef7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 0x7f6f0006c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:41.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:41.594+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 0x7f6f000cc290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:43.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.190+0000 7f6efcd99640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 msgr2=0x7f6f000cc290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f0006d570 0x7f6f000cc290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 -- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 msgr2=0x7f6f0006c5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6f0015b940 0x7f6f0006c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 -- >> v1:172.21.15.157:6789/0 conn(0x7f6f0015d4f0 legacy=0x7f6f0015a4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 0x7f6ee0009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0006c150 con 0x7f6ee0003690 2024-03-23T15:46:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efcd99640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6ee0007370 2024-03-23T15:46:43.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6efe59c640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee0003690 legacy=0x7f6ee0005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:43.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6ef7fff640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:43.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.191+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:43.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.392+0000 7f6efe59c640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee0003690 legacy=0x7f6ee0005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:43.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.392+0000 7f6ef7fff640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:43.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.392+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.793+0000 7f6efe59c640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee0003690 legacy=0x7f6ee0005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.793+0000 7f6ef7fff640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:43.793+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:44.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:44.594+0000 7f6efe59c640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee0003690 legacy=0x7f6ee0005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:44.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:44.594+0000 7f6ef7fff640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:44.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:44.594+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0007370 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee0003690 legacy=0x7f6ee0005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efed9d640 1 -- reap_dead start 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f00154c90 con 0x7f6ee000a330 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6ee0006b80 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efcd99640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0006bb90 con 0x7f6ee0009f00 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6ef7fff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee000a330 legacy=0x7f6ee000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efe59c640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.191+0000 7f6efe59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:46.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.392+0000 7f6ef7fff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee000a330 legacy=0x7f6ee000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:46.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.392+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:46.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.392+0000 7f6efe59c640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:46.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.392+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:46.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.392+0000 7f6efe59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:46.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.793+0000 7f6ef7fff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee000a330 legacy=0x7f6ee000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:46.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.793+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:46.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.793+0000 7f6efe59c640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:46.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.793+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:46.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:46.793+0000 7f6efe59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:46.954 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:00:06.797 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:46:47.262 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:00:07.105 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:46:47.335 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:00:07.133 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:46:47.336 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:00:07.134 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:46:47.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:47.594+0000 7f6ef7fff640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee000a330 legacy=0x7f6ee000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.129:6789/0 2024-03-23T15:46:47.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:47.594+0000 7f6efed9d640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3300/0 2024-03-23T15:46:47.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:47.594+0000 7f6efe59c640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.157:3301/0 2024-03-23T15:46:47.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:47.594+0000 7f6efed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:47.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:47.594+0000 7f6efe59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:47.656 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:00:07.454 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:46:47.658 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:00:07.456 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:46:49.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 -- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:49.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:49.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 -- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 -- >> v1:172.21.15.129:6789/0 conn(0x7f6ee000a330 legacy=0x7f6ee000a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0006c150 con 0x7f6ee000e1a0 2024-03-23T15:46:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.191+0000 7f6efcd99640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6ee000d6a0 2024-03-23T15:46:49.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.209+0000 7f6efe59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:49.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.209+0000 7f6efe59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44734/0 (socket says 172.21.15.129:44734) 2024-03-23T15:46:49.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.209+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 learned_addr learned my addr 172.21.15.129:0/997719642 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:46:49.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.210+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:49.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.210+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:49.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.210+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:49.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.210+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:49.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.449+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:49.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.689+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:49.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.689+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:49.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.689+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:49.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:49.689+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:50.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:50.197+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:50.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:50.201+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:50.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:50.201+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:50.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:50.201+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:50.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:50.201+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:51.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:51.003+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:51.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:51.004+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:51.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:51.004+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:51.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:51.004+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:51.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:51.004+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:52.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 msgr2=0x7f6ee000dab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:52.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee000d6a0 0x7f6ee000dab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:52.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 >> v1:172.21.15.157:6789/0 conn(0x7f6ee000e1a0 legacy=0x7f6ee00127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:52.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efed9d640 1 -- 172.21.15.129:0/997719642 reap_dead start 2024-03-23T15:46:52.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:52.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:52.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f00154c90 con 0x7f6ee000a330 2024-03-23T15:46:52.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6ee0006b80 2024-03-23T15:46:52.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0006bb90 con 0x7f6ee0009f00 2024-03-23T15:46:52.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.192+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:52.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.193+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:52.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.193+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:52.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.193+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:52.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.193+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:52.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.193+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:52.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.196+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T15:46:52.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.196+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:52.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.196+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:52.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.196+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.395+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:52.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.396+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:52.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.396+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:52.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.396+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:52.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.396+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:52.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.397+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.399+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.399+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.399+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.399+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:52.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.797+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.799+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.799+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.799+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.799+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:52.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.800+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.801+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.801+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.801+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:52.802+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:53.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.601+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:53.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.602+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:53.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.602+0000 7f6efe59c640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:53.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.602+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:53.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.602+0000 7f6efe59c640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.603+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:53.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.604+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T15:46:53.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.604+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:53.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.604+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:53.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:53.604+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:55.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 msgr2=0x7f6ee0009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ee0009f00 0x7f6ee0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 msgr2=0x7f6ee000d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.157:3300/0 conn(0x7f6ee0006b80 0x7f6ee000d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 >> v1:172.21.15.129:6789/0 conn(0x7f6ee000a330 legacy=0x7f6ee0013f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee000db00 con 0x7f6ee000a830 2024-03-23T15:46:55.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.191+0000 7f6efcd99640 1 -- 172.21.15.129:0/997719642 --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6ee00117c0 2024-03-23T15:46:55.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.276+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:55.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.277+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T15:46:55.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.277+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:55.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.277+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:55.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:55.277+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:46:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.171+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.172+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T15:46:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.172+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.172+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.172+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:46:56.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.580+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:56.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.583+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T15:46:56.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.583+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:56.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.583+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:56.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:56.583+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:46:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.409+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.454+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T15:46:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.454+0000 7f6ef7fff640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T15:46:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.454+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T15:46:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.454+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:46:57.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.507+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3276988489 0 0) 0x7f6ee000db00 con 0x7f6ee000a830 2024-03-23T15:46:57.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.507+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eec003520 con 0x7f6ee000a830 2024-03-23T15:46:57.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2273723564 0 0) 0x7f6eec003520 con 0x7f6ee000a830 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 msgr2=0x7f6ee0014f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 --2- 172.21.15.129:0/997719642 >> v2:172.21.15.129:3300/0 conn(0x7f6ee00117c0 0x7f6ee0014f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee000ac10 con 0x7f6ee000a830 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f6ef0009770 con 0x7f6ee000a830 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 >> v1:172.21.15.157:6789/0 conn(0x7f6ee000a830 legacy=0x7f6ee0014690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efed9d640 1 -- 172.21.15.129:0/997719642 reap_dead start 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0039a0 0x7f6eec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0047c0 0x7f6eec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0051d0 0x7f6eec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0006cb20 con 0x7f6eec0047c0 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec00c120 con 0x7f6eec0039a0 2024-03-23T15:46:57.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.623+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6eec00c2e0 con 0x7f6eec0051d0 2024-03-23T15:46:57.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.626+0000 7f6efed9d640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0051d0 0x7f6eec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:57.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.630+0000 7f6efed9d640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0039a0 msgr2=0x7f6eec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:46:57.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.630+0000 7f6efed9d640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0039a0 0x7f6eec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.630+0000 7f6efed9d640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0047c0 msgr2=0x7f6eec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:46:57.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.630+0000 7f6efed9d640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0047c0 0x7f6eec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.630+0000 7f6efed9d640 1 -- 172.21.15.129:0/997719642 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec003520 con 0x7f6eec0051d0 2024-03-23T15:46:57.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.632+0000 7f6efed9d640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0051d0 0x7f6eec00b980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b2620 tx=0x7f6eec0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=986431c75271e207 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:57.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.670+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef802d680 con 0x7f6eec0051d0 2024-03-23T15:46:57.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.670+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef8009030 con 0x7f6eec0051d0 2024-03-23T15:46:57.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.670+0000 7f6efdd9b640 1 -- 172.21.15.129:0/997719642 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef802daa0 con 0x7f6eec0051d0 2024-03-23T15:46:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 -- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0051d0 msgr2=0x7f6eec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0051d0 0x7f6eec00b980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b2620 tx=0x7f6eec0057f0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 -- 172.21.15.129:0/997719642 shutdown_connections 2024-03-23T15:46:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0051d0 0x7f6eec00b980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0047c0 0x7f6eec004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 --2- 172.21.15.129:0/997719642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0039a0 0x7f6eec003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 -- 172.21.15.129:0/997719642 >> 172.21.15.129:0/997719642 conn(0x7f6f000cfac0 msgr2=0x7f6f000cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:57.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 -- 172.21.15.129:0/997719642 shutdown_connections 2024-03-23T15:46:57.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.671+0000 7f6f04815640 1 -- 172.21.15.129:0/997719642 wait complete. 2024-03-23T15:46:57.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.672+0000 7f6f04815640 1 Processor -- start 2024-03-23T15:46:57.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.672+0000 7f6f04815640 1 -- start start 2024-03-23T15:46:57.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.673+0000 7f6f04815640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0039a0 0x7f6f000c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:57.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.673+0000 7f6f04815640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 0x7f6f0014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.673+0000 7f6f04815640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0051d0 0x7f6f0014e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.673+0000 7f6f04815640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0006f660 con 0x7f6eec0051d0 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.673+0000 7f6f04815640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00156430 con 0x7f6eec0047c0 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.673+0000 7f6f04815640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f00070500 con 0x7f6eec0039a0 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.674+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 0x7f6f0014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.674+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 0x7f6f0014ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48636/0 (socket says 172.21.15.129:48636) 2024-03-23T15:46:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.674+0000 7f6ef7fff640 1 -- 172.21.15.129:0/519829904 learned_addr learned my addr 172.21.15.129:0/519829904 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:46:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.683+0000 7f6efe59c640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0039a0 0x7f6f000c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:57.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.686+0000 7f6ef7fff640 1 -- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0039a0 msgr2=0x7f6f000c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:57.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.686+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0039a0 0x7f6f000c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.686+0000 7f6ef7fff640 1 -- 172.21.15.129:0/519829904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0051d0 msgr2=0x7f6f0014e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:57.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.686+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0051d0 0x7f6f0014e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:57.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.686+0000 7f6ef7fff640 1 -- 172.21.15.129:0/519829904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f000c6810 con 0x7f6eec0047c0 2024-03-23T15:46:57.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.689+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 0x7f6f0014ddb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8000ea0 tx=0x7f6ee801b3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.698+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee801bb50 con 0x7f6eec0047c0 2024-03-23T15:46:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.698+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee8020070 con 0x7f6eec0047c0 2024-03-23T15:46:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.698+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee801c440 con 0x7f6eec0047c0 2024-03-23T15:46:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.698+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f000c40a0 con 0x7f6eec0047c0 2024-03-23T15:46:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.699+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f000c4660 con 0x7f6eec0047c0 2024-03-23T15:46:57.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.699+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f000ce2e0 con 0x7f6eec0047c0 2024-03-23T15:46:57.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.888+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6ee8022030 con 0x7f6eec0047c0 2024-03-23T15:46:57.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.897+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6ee802ea10 con 0x7f6eec0047c0 2024-03-23T15:46:57.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.897+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6ee802e440 con 0x7f6eec0047c0 2024-03-23T15:46:57.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:57.960+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6f00075740 con 0x7f6eec0047c0 2024-03-23T15:46:58.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.203+0000 7f6ef6ffd640 1 -- 172.21.15.129:0/519829904 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6ee802ebd0 con 0x7f6eec0047c0 2024-03-23T15:46:58.206 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-03-23T15:46:58.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 msgr2=0x7f6f0014ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:58.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 0x7f6f0014ddb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8000ea0 tx=0x7f6ee801b3d0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 shutdown_connections 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6eec0051d0 0x7f6f0014e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eec0047c0 0x7f6f0014ddb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 --2- 172.21.15.129:0/519829904 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eec0039a0 0x7f6f000c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 >> 172.21.15.129:0/519829904 conn(0x7f6f000cfac0 msgr2=0x7f6f000cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 shutdown_connections 2024-03-23T15:46:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:46:58.204+0000 7f6f04815640 1 -- 172.21.15.129:0/519829904 wait complete. 2024-03-23T15:46:58.212 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:58.209+0000 7fbb1d63e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T15:46:58.218 INFO:tasks.ceph:check_enable_crimson: False 2024-03-23T15:46:58.218 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-23T15:46:58.218 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:58.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-23T15:46:58.247 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:58.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-23T15:46:58.269 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:58.266+0000 7fbb1d63e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T15:46:58.315 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:58.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-23T15:46:58.383 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-23T15:46:58.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-23T15:46:58.449 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:58.449 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-23T15:46:58.476 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:58.476 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-23T15:46:58.543 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:58.543 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-23T15:46:58.611 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-23T15:46:58.611 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-23T15:46:58.678 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 4924c046-2ce0-43d1-a4e5-d8d53ba55590 0 2024-03-23T15:46:58.735 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:58.733+0000 7f8cae642040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T15:46:58.789 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:58.787+0000 7f8cae642040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T15:46:58.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.818+0000 7fc631577640 1 Processor -- start 2024-03-23T15:46:58.821 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.818+0000 7fc631577640 1 -- start start 2024-03-23T15:46:58.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.819+0000 7fc631577640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc62c151420 0x7fc62c15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:58.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.819+0000 7fc631577640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c151800 con 0x7fc62c0c7180 2024-03-23T15:46:58.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.819+0000 7fc631577640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c15a640 con 0x7fc62c061be0 2024-03-23T15:46:58.822 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.819+0000 7fc631577640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc62c0c1d10 con 0x7fc62c151420 2024-03-23T15:46:58.823 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.819+0000 7fc62a7fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc62c061be0 0x7fc62c072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:35638/0 (socket says 172.21.15.157:35638) 2024-03-23T15:46:58.823 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.819+0000 7fc62a7fc640 1 -- 172.21.15.157:0/2390650516 learned_addr learned my addr 172.21.15.157:0/2390650516 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-23T15:46:58.823 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.820+0000 7fc62affd640 1 --2- 172.21.15.157:0/2390650516 >> v2:172.21.15.129:3300/0 conn(0x7fc62c151420 0x7fc62c15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:58.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.822+0000 7fc629ffb640 1 -- 172.21.15.157:0/2390650516 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177474351 0 0) 0x7fc62c15a640 con 0x7fc62c061be0 2024-03-23T15:46:58.825 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.822+0000 7fc629ffb640 1 -- 172.21.15.157:0/2390650516 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc614003520 con 0x7fc62c061be0 2024-03-23T15:46:58.825 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:58.823+0000 7fbb1d63e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T15:46:58.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.823+0000 7fc62affd640 1 -- 172.21.15.157:0/2390650516 >> v1:172.21.15.157:6789/0 conn(0x7fc62c061be0 legacy=0x7fc62c072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:58.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.823+0000 7fc62affd640 1 -- 172.21.15.157:0/2390650516 >> v1:172.21.15.129:6789/0 conn(0x7fc62c0c7180 legacy=0x7fc62c161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:58.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.823+0000 7fc62affd640 1 -- 172.21.15.157:0/2390650516 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c06b4c0 con 0x7fc62c151420 2024-03-23T15:46:58.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.830+0000 7fc62affd640 1 --2- 172.21.15.157:0/2390650516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c151420 0x7fc62c15ae00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc62c06b6b0 tx=0x7fc62002b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a83f9f5f30fe26e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:58.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc629ffb640 1 -- 172.21.15.157:0/2390650516 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620030890 con 0x7fc62c151420 2024-03-23T15:46:58.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc629ffb640 1 -- 172.21.15.157:0/2390650516 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6200309f0 con 0x7fc62c151420 2024-03-23T15:46:58.838 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc629ffb640 1 -- 172.21.15.157:0/2390650516 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620030d80 con 0x7fc62c151420 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 -- 172.21.15.157:0/2390650516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c151420 msgr2=0x7fc62c15ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 --2- 172.21.15.157:0/2390650516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c151420 0x7fc62c15ae00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc62c06b6b0 tx=0x7fc62002b9d0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 -- 172.21.15.157:0/2390650516 shutdown_connections 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 --2- 172.21.15.157:0/2390650516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c151420 0x7fc62c15ae00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 -- 172.21.15.157:0/2390650516 >> 172.21.15.157:0/2390650516 conn(0x7fc62c06e490 msgr2=0x7fc62c06f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 -- 172.21.15.157:0/2390650516 shutdown_connections 2024-03-23T15:46:58.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.835+0000 7fc631577640 1 -- 172.21.15.157:0/2390650516 wait complete. 2024-03-23T15:46:58.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 Processor -- start 2024-03-23T15:46:58.840 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 -- start start 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc62c0c7180 0x7fc62c0cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc62c151420 0x7fc62c1e1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 0x7fc62c0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c1e1d80 con 0x7fc62c14da50 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c1ef2c0 con 0x7fc62c0c7180 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.837+0000 7fc631577640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc62c1519a0 con 0x7fc62c151420 2024-03-23T15:46:58.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.838+0000 7fc62affd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc62c0c7180 0x7fc62c0cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:58.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.838+0000 7fc62b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 0x7fc62c0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:58.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.838+0000 7fc62a7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc62c151420 0x7fc62c1e1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:58.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.838+0000 7fc62b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 0x7fc62c0ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:47956/0 (socket says 172.21.15.157:47956) 2024-03-23T15:46:58.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.838+0000 7fc62b7fe640 1 -- 172.21.15.157:0/1904156139 learned_addr learned my addr 172.21.15.157:0/1904156139 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:46:58.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.842+0000 7fc62b7fe640 1 -- 172.21.15.157:0/1904156139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc62c151420 msgr2=0x7fc62c1e1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:58.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.842+0000 7fc62b7fe640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc62c151420 0x7fc62c1e1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.842+0000 7fc62b7fe640 1 -- 172.21.15.157:0/1904156139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc62c0c7180 msgr2=0x7fc62c0cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:58.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.842+0000 7fc62b7fe640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc62c0c7180 0x7fc62c0cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:58.845 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.842+0000 7fc62b7fe640 1 -- 172.21.15.157:0/1904156139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc620009400 con 0x7fc62c14da50 2024-03-23T15:46:58.847 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.844+0000 7fc62b7fe640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 0x7fc62c0ca6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc62400c7a0 tx=0x7fc62400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:58.851 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.848+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624003b70 con 0x7fc62c14da50 2024-03-23T15:46:58.851 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.848+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc624003cf0 con 0x7fc62c14da50 2024-03-23T15:46:58.851 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.848+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624003ea0 con 0x7fc62c14da50 2024-03-23T15:46:58.851 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.848+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62c151c80 con 0x7fc62c14da50 2024-03-23T15:46:58.852 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.848+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62c0c9ec0 con 0x7fc62c14da50 2024-03-23T15:46:58.852 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.849+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc62c062360 con 0x7fc62c14da50 2024-03-23T15:46:58.854 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.851+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc62400aa90 con 0x7fc62c14da50 2024-03-23T15:46:58.883 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:58.880+0000 7fbb1d63e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T15:46:58.929 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:58.926+0000 7fbb1d63e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T15:46:58.968 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.964+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fc6240257c0 con 0x7fc62c14da50 2024-03-23T15:46:58.968 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:58.964+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc62c151800 con 0x7fc62c14da50 2024-03-23T15:46:58.985 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:58.982+0000 7fbb1d63e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T15:46:59.029 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.026+0000 7fbb1d63e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T15:46:59.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.037+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4924c046-2ce0-43d1-a4e5-d8d53ba55590", "id": 0} v 0) -- 0x7fc62c075740 con 0x7fc62c14da50 2024-03-23T15:46:59.148 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.145+0000 7fc6137fe640 1 -- 172.21.15.157:0/1904156139 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4924c046-2ce0-43d1-a4e5-d8d53ba55590", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fc624025980 con 0x7fc62c14da50 2024-03-23T15:46:59.148 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.145+0000 7fbb1d63e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T15:46:59.149 INFO:teuthology.orchestra.run.smithi157.stdout:0 2024-03-23T15:46:59.150 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 msgr2=0x7fc62c0ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.150 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 0x7fc62c0ca6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc62400c7a0 tx=0x7fc62400cc60 comp rx=0 tx=0).stop 2024-03-23T15:46:59.150 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 shutdown_connections 2024-03-23T15:46:59.150 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc62c14da50 0x7fc62c0ca6c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.150 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc62c151420 0x7fc62c1e1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.150 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 --2- 172.21.15.157:0/1904156139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc62c0c7180 0x7fc62c0cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.151 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 >> 172.21.15.157:0/1904156139 conn(0x7fc62c06e490 msgr2=0x7fc62c14e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:59.151 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 shutdown_connections 2024-03-23T15:46:59.151 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.146+0000 7fc631577640 1 -- 172.21.15.157:0/1904156139 wait complete. 2024-03-23T15:46:59.161 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 59cec624-7f42-4a23-a75a-ee39f84bac23 1 2024-03-23T15:46:59.225 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.222+0000 7fbb1d63e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T15:46:59.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.260+0000 7f24be447640 1 Processor -- start 2024-03-23T15:46:59.264 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.260+0000 7f24be447640 1 -- start start 2024-03-23T15:46:59.264 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.261+0000 7f24be447640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 0x7f24b806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.261+0000 7f24be447640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f24b815b940 0x7f24b815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.261+0000 7f24be447640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b8154c90 con 0x7f24b806d570 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.261+0000 7f24be447640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f24b806cb20 con 0x7f24b815b940 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.261+0000 7f24be447640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24b806bb90 con 0x7f24b815d4f0 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.261+0000 7f24b7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 0x7f24b806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.262+0000 7f24b7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 0x7f24b806c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:55364/0 (socket says 172.21.15.157:55364) 2024-03-23T15:46:59.265 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.262+0000 7f24b7fff640 1 -- 172.21.15.157:0/3205065850 learned_addr learned my addr 172.21.15.157:0/3205065850 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:46:59.266 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.262+0000 7f24b77fe640 1 --2- 172.21.15.157:0/3205065850 >> v2:172.21.15.129:3300/0 conn(0x7f24b815b940 0x7f24b815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.267 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.264+0000 7f24b7fff640 1 -- 172.21.15.157:0/3205065850 >> v2:172.21.15.129:3300/0 conn(0x7f24b815b940 msgr2=0x7f24b815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:46:59.267 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.264+0000 7f24b7fff640 1 --2- 172.21.15.157:0/3205065850 >> v2:172.21.15.129:3300/0 conn(0x7f24b815b940 0x7f24b815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.267 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.264+0000 7f24b7fff640 1 -- 172.21.15.157:0/3205065850 >> v1:172.21.15.129:6789/0 conn(0x7f24b806d570 legacy=0x7f24b80cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.268 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.264+0000 7f24b7fff640 1 -- 172.21.15.157:0/3205065850 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b806c150 con 0x7f24b815d4f0 2024-03-23T15:46:59.269 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.266+0000 7f24b7fff640 1 --2- 172.21.15.157:0/3205065850 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 0x7f24b806c5e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009870 tx=0x7f24a802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77f2d3716e65f697 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:59.273 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.270+0000 7f24b6ffd640 1 -- 172.21.15.157:0/3205065850 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a802f840 con 0x7f24b815d4f0 2024-03-23T15:46:59.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.270+0000 7f24b6ffd640 1 -- 172.21.15.157:0/3205065850 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a802f9a0 con 0x7f24b815d4f0 2024-03-23T15:46:59.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.270+0000 7f24b6ffd640 1 -- 172.21.15.157:0/3205065850 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a802fd30 con 0x7f24b815d4f0 2024-03-23T15:46:59.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 -- 172.21.15.157:0/3205065850 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 msgr2=0x7f24b806c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 --2- 172.21.15.157:0/3205065850 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 0x7f24b806c5e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009870 tx=0x7f24a802ba20 comp rx=0 tx=0).stop 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 -- 172.21.15.157:0/3205065850 shutdown_connections 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 --2- 172.21.15.157:0/3205065850 >> v2:172.21.15.129:3300/0 conn(0x7f24b815b940 0x7f24b815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 --2- 172.21.15.157:0/3205065850 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815d4f0 0x7f24b806c5e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 -- 172.21.15.157:0/3205065850 >> 172.21.15.157:0/3205065850 conn(0x7f24b80cfac0 msgr2=0x7f24b80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 -- 172.21.15.157:0/3205065850 shutdown_connections 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.271+0000 7f24be447640 1 -- 172.21.15.157:0/3205065850 wait complete. 2024-03-23T15:46:59.275 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.272+0000 7f24be447640 1 Processor -- start 2024-03-23T15:46:59.276 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 -- start start 2024-03-23T15:46:59.276 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.273+0000 7fbb1d63e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T15:46:59.276 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 0x7f24b81f1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.276 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815b940 0x7f24b814f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b815d4f0 0x7f24b80c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f24b814f030 con 0x7f24b815d4f0 2024-03-23T15:46:59.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81f0940 con 0x7f24b806d570 2024-03-23T15:46:59.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.273+0000 7f24be447640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24b8150650 con 0x7f24b815b940 2024-03-23T15:46:59.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.274+0000 7f24b7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 0x7f24b81f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.274+0000 7f24b77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815b940 0x7f24b814f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.278 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.274+0000 7f24b7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 0x7f24b81f1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:48014/0 (socket says 172.21.15.157:48014) 2024-03-23T15:46:59.278 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.274+0000 7f24b7fff640 1 -- 172.21.15.157:0/4078533771 learned_addr learned my addr 172.21.15.157:0/4078533771 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:46:59.278 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.275+0000 7f24bc9e4640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b815d4f0 0x7f24b80c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.278+0000 7f24b7fff640 1 -- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815b940 msgr2=0x7f24b814f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.278+0000 7f24b7fff640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815b940 0x7f24b814f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.278+0000 7f24b7fff640 1 -- 172.21.15.157:0/4078533771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b815d4f0 msgr2=0x7f24b80c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.278+0000 7f24b7fff640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b815d4f0 0x7f24b80c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.278+0000 7f24b7fff640 1 -- 172.21.15.157:0/4078533771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a80094d0 con 0x7f24b806d570 2024-03-23T15:46:59.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.280+0000 7f24b7fff640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 0x7f24b81f1030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f24a8000c00 tx=0x7f24a80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:59.287 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.284+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a803b040 con 0x7f24b806d570 2024-03-23T15:46:59.287 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.284+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a8037c60 con 0x7f24b806d570 2024-03-23T15:46:59.287 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.284+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8042480 con 0x7f24b806d570 2024-03-23T15:46:59.287 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.284+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b8150910 con 0x7f24b806d570 2024-03-23T15:46:59.287 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.284+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b814ec50 con 0x7f24b806d570 2024-03-23T15:46:59.288 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.285+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b814f170 con 0x7f24b806d570 2024-03-23T15:46:59.291 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.288+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24a80425e0 con 0x7f24b806d570 2024-03-23T15:46:59.291 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.288+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f24a804cd50 con 0x7f24b806d570 2024-03-23T15:46:59.331 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.328+0000 7f8cae642040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T15:46:59.332 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.329+0000 7fbb1d63e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T15:46:59.387 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.379+0000 7fbb1d63e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T15:46:59.387 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.383+0000 7f8cae642040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T15:46:59.388 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.385+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f24b814f170 con 0x7f24b806d570 2024-03-23T15:46:59.435 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.432+0000 7f8cae642040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T15:46:59.480 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.476+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "59cec624-7f42-4a23-a75a-ee39f84bac23", "id": 1} v 0) -- 0x7f24b8075740 con 0x7f24b806d570 2024-03-23T15:46:59.489 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.486+0000 7f8cae642040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T15:46:59.533 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.530+0000 7f8cae642040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T15:46:59.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.571+0000 7f24b4ff9640 1 -- 172.21.15.157:0/4078533771 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "59cec624-7f42-4a23-a75a-ee39f84bac23", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f24a804c460 con 0x7f24b806d570 2024-03-23T15:46:59.574 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-03-23T15:46:59.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 msgr2=0x7f24b81f1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 0x7f24b81f1030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f24a8000c00 tx=0x7f24a80046c0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 shutdown_connections 2024-03-23T15:46:59.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b815d4f0 0x7f24b80c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24b815b940 0x7f24b814f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 --2- 172.21.15.157:0/4078533771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b806d570 0x7f24b81f1030 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 >> 172.21.15.157:0/4078533771 conn(0x7f24b80cfac0 msgr2=0x7f24b8158210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:59.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 shutdown_connections 2024-03-23T15:46:59.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.572+0000 7f24be447640 1 -- 172.21.15.157:0/4078533771 wait complete. 2024-03-23T15:46:59.586 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new b246bd79-82a4-4808-82cc-c78d286260ff 2 2024-03-23T15:46:59.650 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.648+0000 7f8cae642040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T15:46:59.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.688+0000 7f81de48f640 1 Processor -- start 2024-03-23T15:46:59.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.689+0000 7f81de48f640 1 -- start start 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.689+0000 7f81de48f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d8072c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.689+0000 7f81de48f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f81d80c8bb0 0x7f81d80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.689+0000 7f81de48f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d814e0f0 con 0x7f81d80ccc10 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.689+0000 7f81de48f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f81d80c5140 con 0x7f81d80c8bb0 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.689+0000 7f81de48f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81d80c9210 con 0x7f81d814dd10 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.690+0000 7f81dca2c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f81d80ccc10 0x7f81d80c7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.157:37118/0 (socket says 172.21.15.157:37118) 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.690+0000 7f81dca2c640 1 -- 172.21.15.157:0/454829719 learned_addr learned my addr 172.21.15.157:0/454829719 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-23T15:46:59.693 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.690+0000 7f81d77fe640 1 --2- 172.21.15.157:0/454829719 >> v2:172.21.15.157:3300/0 conn(0x7f81d80c8bb0 0x7f81d80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.690+0000 7f81d7fff640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d8072c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.692+0000 7f81d77fe640 1 -- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 msgr2=0x7f81d8072c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:46:59.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.692+0000 7f81d77fe640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d8072c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.692+0000 7f81d77fe640 1 -- 172.21.15.157:0/454829719 >> v1:172.21.15.129:6789/0 conn(0x7f81d80ccc10 legacy=0x7f81d80c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.693+0000 7f81d77fe640 1 -- 172.21.15.157:0/454829719 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d80c94f0 con 0x7f81d80c8bb0 2024-03-23T15:46:59.696 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.693+0000 7f81d7fff640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d8072c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:46:59.697 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.695+0000 7f81d77fe640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d80cc6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f81c80097a0 tx=0x7f81c802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3509c550d4cc361e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:59.701 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.698+0000 7f81d6ffd640 1 -- 172.21.15.157:0/454829719 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8031830 con 0x7f81d80c8bb0 2024-03-23T15:46:59.701 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.698+0000 7f81d6ffd640 1 -- 172.21.15.157:0/454829719 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c8031990 con 0x7f81d80c8bb0 2024-03-23T15:46:59.701 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.698+0000 7f81d6ffd640 1 -- 172.21.15.157:0/454829719 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8031d20 con 0x7f81d80c8bb0 2024-03-23T15:46:59.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 -- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 msgr2=0x7f81d80cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d80cc6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f81c80097a0 tx=0x7f81c802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 -- 172.21.15.157:0/454829719 shutdown_connections 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d80cc6d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 --2- 172.21.15.157:0/454829719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d8072c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 -- 172.21.15.157:0/454829719 >> 172.21.15.157:0/454829719 conn(0x7f81d8062180 msgr2=0x7f81d8070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 -- 172.21.15.157:0/454829719 shutdown_connections 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.699+0000 7f81de48f640 1 -- 172.21.15.157:0/454829719 wait complete. 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.700+0000 7f81de48f640 1 Processor -- start 2024-03-23T15:46:59.703 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 -- start start 2024-03-23T15:46:59.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d8142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 0x7f81d81435e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d80c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:46:59.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f81d814e930 con 0x7f81d80ccc10 2024-03-23T15:46:59.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81d8142110 con 0x7f81d80c8bb0 2024-03-23T15:46:59.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.701+0000 7f81de48f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81d81423f0 con 0x7f81d814dd10 2024-03-23T15:46:59.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.702+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 0x7f81d81435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.702+0000 7f81dca2c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d80c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.702+0000 7f81d7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d8142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:46:59.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.702+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 0x7f81d81435e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:47976/0 (socket says 172.21.15.157:47976) 2024-03-23T15:46:59.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.702+0000 7f81d77fe640 1 -- 172.21.15.157:0/3989296158 learned_addr learned my addr 172.21.15.157:0/3989296158 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:46:59.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.705+0000 7f81d77fe640 1 -- 172.21.15.157:0/3989296158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 msgr2=0x7f81d80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.705+0000 7f81d77fe640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d80c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.705+0000 7f81d77fe640 1 -- 172.21.15.157:0/3989296158 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 msgr2=0x7f81d8142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.708 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.705+0000 7f81d77fe640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d8142660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.709 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.705+0000 7f81d77fe640 1 -- 172.21.15.157:0/3989296158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc009400 con 0x7f81d80ccc10 2024-03-23T15:46:59.710 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.707+0000 7f81d77fe640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 0x7f81d81435e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f81c8034040 tx=0x7f81c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:46:59.714 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.711+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c803a8f0 con 0x7f81d80ccc10 2024-03-23T15:46:59.714 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.711+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c803aa70 con 0x7f81d80ccc10 2024-03-23T15:46:59.714 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.711+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c8009400 con 0x7f81d80ccc10 2024-03-23T15:46:59.714 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.711+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c803ae20 con 0x7f81d80ccc10 2024-03-23T15:46:59.714 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.711+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d814e650 con 0x7f81d80ccc10 2024-03-23T15:46:59.715 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.712+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d8075740 con 0x7f81d80ccc10 2024-03-23T15:46:59.721 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.718+0000 7f8cae642040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T15:46:59.768 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.766+0000 7f8cae642040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T15:46:59.791 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.788+0000 7fbb1d63e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T15:46:59.810 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.806+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f81c804d7d0 con 0x7f81d80ccc10 2024-03-23T15:46:59.813 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.810+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f81c80307a0 con 0x7f81d80ccc10 2024-03-23T15:46:59.813 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.810+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f81c8035020 con 0x7f81d80ccc10 2024-03-23T15:46:59.822 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.820+0000 7f8cae642040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T15:46:59.869 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:46:59.866+0000 7f8cae642040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T15:46:59.885 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.882+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b246bd79-82a4-4808-82cc-c78d286260ff", "id": 2} v 0) -- 0x7f81d80c4250 con 0x7f81d80ccc10 2024-03-23T15:46:59.914 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.911+0000 7f81d4ff9640 1 -- 172.21.15.157:0/3989296158 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b246bd79-82a4-4808-82cc-c78d286260ff", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f81c8044400 con 0x7f81d80ccc10 2024-03-23T15:46:59.915 INFO:teuthology.orchestra.run.smithi157.stdout:2 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 msgr2=0x7f81d81435e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 0x7f81d81435e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f81c8034040 tx=0x7f81c80046c0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 shutdown_connections 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81d814dd10 0x7f81d80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81d80ccc10 0x7f81d81435e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 --2- 172.21.15.157:0/3989296158 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81d80c8bb0 0x7f81d8142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 >> 172.21.15.157:0/3989296158 conn(0x7f81d8062180 msgr2=0x7f81d8070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 shutdown_connections 2024-03-23T15:46:59.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:46:59.912+0000 7f81de48f640 1 -- 172.21.15.157:0/3989296158 wait complete. 2024-03-23T15:46:59.927 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 9041cd1c-9759-411f-bc7e-182544af5af7 3 2024-03-23T15:46:59.931 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:46:59.928+0000 7fbb1d63e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T15:47:00.028 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.024+0000 7f1047be9640 1 Processor -- start 2024-03-23T15:47:00.028 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.025+0000 7f1047be9640 1 -- start start 2024-03-23T15:47:00.029 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.026+0000 7f1047be9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f104015c110 0x7f10400c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.029 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.026+0000 7f1047be9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f104006fe00 con 0x7f104006f8a0 2024-03-23T15:47:00.029 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.026+0000 7f1047be9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1040070660 con 0x7f10401521e0 2024-03-23T15:47:00.029 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.026+0000 7f1047be9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f1040070940 con 0x7f104015c110 2024-03-23T15:47:00.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.026+0000 7f1045985640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f10401521e0 0x7f10400c8230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:35646/0 (socket says 172.21.15.157:35646) 2024-03-23T15:47:00.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.026+0000 7f1045985640 1 -- 172.21.15.157:0/4183519660 learned_addr learned my addr 172.21.15.157:0/4183519660 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-23T15:47:00.030 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.027+0000 7f1045184640 1 --2- 172.21.15.157:0/4183519660 >> v2:172.21.15.129:3300/0 conn(0x7f104015c110 0x7f10400c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.033 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.030+0000 7f1044983640 1 -- 172.21.15.157:0/4183519660 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4072946663 0 0) 0x7f1040070660 con 0x7f10401521e0 2024-03-23T15:47:00.033 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.030+0000 7f1044983640 1 -- 172.21.15.157:0/4183519660 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f102c003520 con 0x7f10401521e0 2024-03-23T15:47:00.033 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.030+0000 7f1045184640 1 -- 172.21.15.157:0/4183519660 >> v1:172.21.15.157:6789/0 conn(0x7f10401521e0 legacy=0x7f10400c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.034 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.030+0000 7f1045184640 1 -- 172.21.15.157:0/4183519660 >> v1:172.21.15.129:6789/0 conn(0x7f104006f8a0 legacy=0x7f104015b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.034 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.030+0000 7f1045184640 1 -- 172.21.15.157:0/4183519660 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10400c2910 con 0x7f104015c110 2024-03-23T15:47:00.036 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.033+0000 7f1045184640 1 --2- 172.21.15.157:0/4183519660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104015c110 0x7f10400c3110 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f103c0097a0 tx=0x7f103c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7305734f09968010 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:00.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1044983640 1 -- 172.21.15.157:0/4183519660 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c030610 con 0x7f104015c110 2024-03-23T15:47:00.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1044983640 1 -- 172.21.15.157:0/4183519660 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c030770 con 0x7f104015c110 2024-03-23T15:47:00.041 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1044983640 1 -- 172.21.15.157:0/4183519660 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c030b00 con 0x7f104015c110 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1047be9640 1 -- 172.21.15.157:0/4183519660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104015c110 msgr2=0x7f10400c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1047be9640 1 --2- 172.21.15.157:0/4183519660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104015c110 0x7f10400c3110 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f103c0097a0 tx=0x7f103c02b960 comp rx=0 tx=0).stop 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1047be9640 1 -- 172.21.15.157:0/4183519660 shutdown_connections 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1047be9640 1 --2- 172.21.15.157:0/4183519660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104015c110 0x7f10400c3110 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1047be9640 1 -- 172.21.15.157:0/4183519660 >> 172.21.15.157:0/4183519660 conn(0x7f10400718f0 msgr2=0x7f1040061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.038+0000 7f1047be9640 1 -- 172.21.15.157:0/4183519660 shutdown_connections 2024-03-23T15:47:00.042 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.039+0000 7f1047be9640 1 -- 172.21.15.157:0/4183519660 wait complete. 2024-03-23T15:47:00.043 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 Processor -- start 2024-03-23T15:47:00.043 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 -- start start 2024-03-23T15:47:00.044 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 0x7f1040153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.044 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10401521e0 0x7f10401ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.044 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f104015c110 0x7f10401e2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.044 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10401536d0 con 0x7f104006f8a0 2024-03-23T15:47:00.044 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f10401ee800 con 0x7f10401521e0 2024-03-23T15:47:00.044 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.040+0000 7f1047be9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1040158b60 con 0x7f104015c110 2024-03-23T15:47:00.045 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.041+0000 7f1045985640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 0x7f1040153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.045 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.041+0000 7f1045184640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10401521e0 0x7f10401ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.045 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.041+0000 7f1046186640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f104015c110 0x7f10401e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.045 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.041+0000 7f1045985640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 0x7f1040153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:48006/0 (socket says 172.21.15.157:48006) 2024-03-23T15:47:00.045 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.041+0000 7f1045985640 1 -- 172.21.15.157:0/1103528938 learned_addr learned my addr 172.21.15.157:0/1103528938 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:00.047 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.044+0000 7f1045985640 1 -- 172.21.15.157:0/1103528938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f104015c110 msgr2=0x7f10401e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.047 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.044+0000 7f1045985640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f104015c110 0x7f10401e2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.047 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.044+0000 7f1045985640 1 -- 172.21.15.157:0/1103528938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10401521e0 msgr2=0x7f10401ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.048 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.044+0000 7f1045985640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10401521e0 0x7f10401ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.048 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.044+0000 7f1045985640 1 -- 172.21.15.157:0/1103528938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c009400 con 0x7f104006f8a0 2024-03-23T15:47:00.049 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.046+0000 7f1045985640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 0x7f1040153190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f103000c530 tx=0x7f103000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:00.053 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.050+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030005ce0 con 0x7f104006f8a0 2024-03-23T15:47:00.053 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.050+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1030005e40 con 0x7f104006f8a0 2024-03-23T15:47:00.053 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.050+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10300072b0 con 0x7f104006f8a0 2024-03-23T15:47:00.053 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.050+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10401e1400 con 0x7f104006f8a0 2024-03-23T15:47:00.053 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.050+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10401e1ca0 con 0x7f104006f8a0 2024-03-23T15:47:00.054 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.051+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1040075740 con 0x7f104006f8a0 2024-03-23T15:47:00.060 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.057+0000 7fbb1d63e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T15:47:00.105 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.102+0000 7fbb1d63e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T15:47:00.149 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.146+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1030007410 con 0x7f104006f8a0 2024-03-23T15:47:00.154 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.151+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f103001b840 con 0x7f104006f8a0 2024-03-23T15:47:00.154 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.151+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1030017070 con 0x7f104006f8a0 2024-03-23T15:47:00.220 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.217+0000 7fbb1d63e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T15:47:00.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.223+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9041cd1c-9759-411f-bc7e-182544af5af7", "id": 3} v 0) -- 0x7f104015bc30 con 0x7f104006f8a0 2024-03-23T15:47:00.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.254+0000 7f10367fc640 1 -- 172.21.15.157:0/1103528938 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9041cd1c-9759-411f-bc7e-182544af5af7", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f103001b480 con 0x7f104006f8a0 2024-03-23T15:47:00.258 INFO:teuthology.orchestra.run.smithi157.stdout:3 2024-03-23T15:47:00.262 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.256+0000 7f8cae642040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T15:47:00.262 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 msgr2=0x7f1040153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 0x7f1040153190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f103000c530 tx=0x7f103000c9f0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 shutdown_connections 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f104015c110 0x7f10401e2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10401521e0 0x7f10401ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 --2- 172.21.15.157:0/1103528938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f104006f8a0 0x7f1040153190 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.255+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 >> 172.21.15.157:0/1103528938 conn(0x7f10400718f0 msgr2=0x7f10400c4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.256+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 shutdown_connections 2024-03-23T15:47:00.263 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.256+0000 7f1047be9640 1 -- 172.21.15.157:0/1103528938 wait complete. 2024-03-23T15:47:00.270 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 6ae79e67-6ebc-48ac-931d-5057f09c098c 4 2024-03-23T15:47:00.323 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.320+0000 7fbb1d63e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T15:47:00.396 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.393+0000 7f8cae642040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T15:47:00.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.411+0000 7fc585a6f640 1 Processor -- start 2024-03-23T15:47:00.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.411+0000 7fc585a6f640 1 -- start start 2024-03-23T15:47:00.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc585a6f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc58014dd10 0x7fc580072c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc585a6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc5800cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc585a6f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5800ccc10 0x7fc5800c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc585a6f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc5800c5140 con 0x7fc5800ccc10 2024-03-23T15:47:00.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc585a6f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc58014e0f0 con 0x7fc58014dd10 2024-03-23T15:47:00.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc585a6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc5800c9210 con 0x7fc5800c8bb0 2024-03-23T15:47:00.418 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc57effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc5800cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.412+0000 7fc57f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc58014dd10 0x7fc580072c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.413+0000 7fc57f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc58014dd10 0x7fc580072c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:48058/0 (socket says 172.21.15.157:48058) 2024-03-23T15:47:00.419 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.413+0000 7fc57f7fe640 1 -- 172.21.15.157:0/2482391516 learned_addr learned my addr 172.21.15.157:0/2482391516 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:00.420 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.415+0000 7fc57f7fe640 1 -- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 msgr2=0x7fc5800cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:00.420 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.415+0000 7fc57f7fe640 1 --2- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc5800cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.421 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.415+0000 7fc57f7fe640 1 -- 172.21.15.157:0/2482391516 >> v2:172.21.15.129:3300/0 conn(0x7fc5800ccc10 msgr2=0x7fc5800c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:00.421 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.415+0000 7fc57f7fe640 1 --2- 172.21.15.157:0/2482391516 >> v2:172.21.15.129:3300/0 conn(0x7fc5800ccc10 0x7fc5800c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.421 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.415+0000 7fc57f7fe640 1 -- 172.21.15.157:0/2482391516 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5800c94f0 con 0x7fc58014dd10 2024-03-23T15:47:00.421 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.415+0000 7fc57effd640 1 --2- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc5800cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:47:00.421 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.417+0000 7fc57f7fe640 1 --2- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58014dd10 0x7fc580072c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc580070f40 tx=0x7fc57002e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43cb93bac5726f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:00.424 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.421+0000 7fc57e7fc640 1 -- 172.21.15.157:0/2482391516 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57002ea80 con 0x7fc58014dd10 2024-03-23T15:47:00.424 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.421+0000 7fc57e7fc640 1 -- 172.21.15.157:0/2482391516 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc57002ebe0 con 0x7fc58014dd10 2024-03-23T15:47:00.424 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.421+0000 7fc57e7fc640 1 -- 172.21.15.157:0/2482391516 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570009600 con 0x7fc58014dd10 2024-03-23T15:47:00.425 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 -- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58014dd10 msgr2=0x7fc580072c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.425 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 --2- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58014dd10 0x7fc580072c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc580070f40 tx=0x7fc57002e0e0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.425 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 -- 172.21.15.157:0/2482391516 shutdown_connections 2024-03-23T15:47:00.426 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 --2- 172.21.15.157:0/2482391516 >> v2:172.21.15.129:3300/0 conn(0x7fc5800ccc10 0x7fc5800c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.426 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 --2- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc5800cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.426 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 --2- 172.21.15.157:0/2482391516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc58014dd10 0x7fc580072c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.427 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 -- 172.21.15.157:0/2482391516 >> 172.21.15.157:0/2482391516 conn(0x7fc580062180 msgr2=0x7fc580070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:00.427 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 -- 172.21.15.157:0/2482391516 shutdown_connections 2024-03-23T15:47:00.427 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.422+0000 7fc585a6f640 1 -- 172.21.15.157:0/2482391516 wait complete. 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.423+0000 7fc585a6f640 1 Processor -- start 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.423+0000 7fc585a6f640 1 -- start start 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc585a6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc58015a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc585a6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5800ccc10 0x7fc58006c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc585a6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc58014dd10 0x7fc580158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc585a6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801ec3a0 con 0x7fc58014dd10 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc585a6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc58015aa40 con 0x7fc5800ccc10 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc585a6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc58015acb0 con 0x7fc5800c8bb0 2024-03-23T15:47:00.428 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.424+0000 7fc57f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc58015a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:00.429 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.425+0000 7fc57f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc58015a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:55418/0 (socket says 172.21.15.157:55418) 2024-03-23T15:47:00.429 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.425+0000 7fc57f7fe640 1 -- 172.21.15.157:0/1562863553 learned_addr learned my addr 172.21.15.157:0/1562863553 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:00.431 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.428+0000 7fc57f7fe640 1 -- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5800ccc10 msgr2=0x7fc58006c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.428+0000 7fc57f7fe640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5800ccc10 0x7fc58006c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.428+0000 7fc57f7fe640 1 -- 172.21.15.157:0/1562863553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc58014dd10 msgr2=0x7fc580158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.428+0000 7fc57f7fe640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc58014dd10 0x7fc580158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.432 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.428+0000 7fc57f7fe640 1 -- 172.21.15.157:0/1562863553 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc568009400 con 0x7fc5800c8bb0 2024-03-23T15:47:00.435 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.432+0000 7fc57f7fe640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc58015a500 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc570038040 tx=0x7fc570004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:00.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.462+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57000b720 con 0x7fc5800c8bb0 2024-03-23T15:47:00.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.462+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc570009e50 con 0x7fc5800c8bb0 2024-03-23T15:47:00.465 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.462+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5700408a0 con 0x7fc5800c8bb0 2024-03-23T15:47:00.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.462+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57000bd40 con 0x7fc5800c8bb0 2024-03-23T15:47:00.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.462+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5801ebd50 con 0x7fc5800c8bb0 2024-03-23T15:47:00.466 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.463+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc580075740 con 0x7fc5800c8bb0 2024-03-23T15:47:00.496 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.493+0000 7fbb1d63e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T15:47:00.521 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.518+0000 7f8cae642040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T15:47:00.540 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.537+0000 7fbb1d63e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T15:47:00.567 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.565+0000 7f8cae642040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T15:47:00.662 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.658+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc57002ed50 con 0x7fc5800c8bb0 2024-03-23T15:47:00.662 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.658+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fc570041400 con 0x7fc5800c8bb0 2024-03-23T15:47:00.662 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.658+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc570045020 con 0x7fc5800c8bb0 2024-03-23T15:47:00.679 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.676+0000 7f8cae642040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T15:47:00.700 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:00.697+0000 7fbb1d63e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T15:47:00.733 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.729+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6ae79e67-6ebc-48ac-931d-5057f09c098c", "id": 4} v 0) -- 0x7fc5800c4190 con 0x7fc5800c8bb0 2024-03-23T15:47:00.773 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.770+0000 7f8cae642040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T15:47:00.852 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.848+0000 7fc577fff640 1 -- 172.21.15.157:0/1562863553 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ae79e67-6ebc-48ac-931d-5057f09c098c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fc570057020 con 0x7fc5800c8bb0 2024-03-23T15:47:00.852 INFO:teuthology.orchestra.run.smithi157.stdout:4 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 msgr2=0x7fc58015a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc58015a500 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc570038040 tx=0x7fc570004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 shutdown_connections 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc58014dd10 0x7fc580158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5800ccc10 0x7fc58006c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 --2- 172.21.15.157:0/1562863553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5800c8bb0 0x7fc58015a500 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:00.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 >> 172.21.15.157:0/1562863553 conn(0x7fc580062180 msgr2=0x7fc58014c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:00.854 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 shutdown_connections 2024-03-23T15:47:00.854 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:00.849+0000 7fc585a6f640 1 -- 172.21.15.157:0/1562863553 wait complete. 2024-03-23T15:47:00.863 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new f679eb27-e464-459b-ab03-06c91915c3be 5 2024-03-23T15:47:00.932 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.929+0000 7f8cae642040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T15:47:00.974 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:00.972+0000 7f8cae642040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T15:47:01.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.000+0000 7f8cb21a5640 1 Processor -- start 2024-03-23T15:47:01.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.000+0000 7f8cb21a5640 1 -- start start 2024-03-23T15:47:01.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.001+0000 7f8cb21a5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8cac1437d0 0x7f8cac0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.001+0000 7f8cb21a5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8cac070330 0x7f8cac0c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.001+0000 7f8cb21a5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cac0c8770 con 0x7f8cac0c9410 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.001+0000 7f8cb21a5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8cac15e7a0 con 0x7f8cac070330 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.001+0000 7f8cb21a5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8cac06d250 con 0x7f8cac1437d0 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.001+0000 7f8cab7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8cac1437d0 0x7f8cac0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.002+0000 7f8ca2ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8cac070330 0x7f8cac0c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.002+0000 7f8cabfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8cac0c9410 0x7f8cac15d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.157:37132/0 (socket says 172.21.15.157:37132) 2024-03-23T15:47:01.006 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.002+0000 7f8cabfff640 1 -- 172.21.15.157:0/177248230 learned_addr learned my addr 172.21.15.157:0/177248230 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-23T15:47:01.007 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.004+0000 7f8cab7fe640 1 -- 172.21.15.157:0/177248230 >> v2:172.21.15.129:3300/0 conn(0x7f8cac070330 msgr2=0x7f8cac0c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:01.007 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.004+0000 7f8cab7fe640 1 --2- 172.21.15.157:0/177248230 >> v2:172.21.15.129:3300/0 conn(0x7f8cac070330 0x7f8cac0c8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.007 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.004+0000 7f8cab7fe640 1 -- 172.21.15.157:0/177248230 >> v1:172.21.15.129:6789/0 conn(0x7f8cac0c9410 legacy=0x7f8cac15d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.007 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.004+0000 7f8cab7fe640 1 -- 172.21.15.157:0/177248230 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cac06c240 con 0x7f8cac1437d0 2024-03-23T15:47:01.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.006+0000 7f8cab7fe640 1 --2- 172.21.15.157:0/177248230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac0c8230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8cac062560 tx=0x7f8c9c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=981a382162514fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:01.012 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.009+0000 7f8caaffd640 1 -- 172.21.15.157:0/177248230 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c02e070 con 0x7f8cac1437d0 2024-03-23T15:47:01.012 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.009+0000 7f8caaffd640 1 -- 172.21.15.157:0/177248230 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9c0315a0 con 0x7f8cac1437d0 2024-03-23T15:47:01.013 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.009+0000 7f8caaffd640 1 -- 172.21.15.157:0/177248230 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c031930 con 0x7f8cac1437d0 2024-03-23T15:47:01.013 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 -- 172.21.15.157:0/177248230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 msgr2=0x7f8cac0c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/177248230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac0c8230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8cac062560 tx=0x7f8c9c02b840 comp rx=0 tx=0).stop 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 -- 172.21.15.157:0/177248230 shutdown_connections 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/177248230 >> v2:172.21.15.129:3300/0 conn(0x7f8cac070330 0x7f8cac0c8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/177248230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac0c8230 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 -- 172.21.15.157:0/177248230 >> 172.21.15.157:0/177248230 conn(0x7f8cac0718f0 msgr2=0x7f8cac061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 -- 172.21.15.157:0/177248230 shutdown_connections 2024-03-23T15:47:01.014 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.010+0000 7f8cb21a5640 1 -- 172.21.15.157:0/177248230 wait complete. 2024-03-23T15:47:01.018 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.011+0000 7f8cb21a5640 1 Processor -- start 2024-03-23T15:47:01.018 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 -- start start 2024-03-23T15:47:01.018 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cac070330 0x7f8cac1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.018 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 0x7f8cac15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1f3ba0 con 0x7f8cac0c9410 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1f4570 con 0x7f8cac1437d0 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.012+0000 7f8cb21a5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8cac0cca40 con 0x7f8cac070330 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.013+0000 7f8cabfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.013+0000 7f8ca2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 0x7f8cac15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.013+0000 7f8cab7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cac070330 0x7f8cac1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.013+0000 7f8ca2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 0x7f8cac15d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:48034/0 (socket says 172.21.15.157:48034) 2024-03-23T15:47:01.019 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.013+0000 7f8ca2ffd640 1 -- 172.21.15.157:0/915770166 learned_addr learned my addr 172.21.15.157:0/915770166 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:01.020 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.016+0000 7f8ca2ffd640 1 -- 172.21.15.157:0/915770166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cac070330 msgr2=0x7f8cac1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.020 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.016+0000 7f8ca2ffd640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cac070330 0x7f8cac1f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.020 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.016+0000 7f8ca2ffd640 1 -- 172.21.15.157:0/915770166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 msgr2=0x7f8cac15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.020 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.016+0000 7f8ca2ffd640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac15c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.020 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.016+0000 7f8ca2ffd640 1 -- 172.21.15.157:0/915770166 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c009400 con 0x7f8cac0c9410 2024-03-23T15:47:01.021 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.018+0000 7f8ca2ffd640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 0x7f8cac15d2b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400e9b0 tx=0x7f8c9400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:01.025 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.022+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94009670 con 0x7f8cac0c9410 2024-03-23T15:47:01.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.022+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9400c9a0 con 0x7f8cac0c9410 2024-03-23T15:47:01.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.022+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9400cd50 con 0x7f8cac0c9410 2024-03-23T15:47:01.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.022+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cac0ccd20 con 0x7f8cac0c9410 2024-03-23T15:47:01.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.022+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cac1f3830 con 0x7f8cac0c9410 2024-03-23T15:47:01.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.023+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cac154000 con 0x7f8cac0c9410 2024-03-23T15:47:01.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.024+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8c94005990 con 0x7f8cac0c9410 2024-03-23T15:47:01.126 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.123+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f8c940061c0 con 0x7f8cac0c9410 2024-03-23T15:47:01.127 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.123+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8c94021060 con 0x7f8cac0c9410 2024-03-23T15:47:01.134 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:01.132+0000 7f8cae642040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T15:47:01.202 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.198+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f679eb27-e464-459b-ab03-06c91915c3be", "id": 5} v 0) -- 0x7f8cac1f3690 con 0x7f8cac0c9410 2024-03-23T15:47:01.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.226+0000 7f8ca8ff9640 1 -- 172.21.15.157:0/915770166 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f679eb27-e464-459b-ab03-06c91915c3be", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f8cac1f3690 con 0x7f8cac0c9410 2024-03-23T15:47:01.230 INFO:teuthology.orchestra.run.smithi157.stdout:5 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 msgr2=0x7f8cac15d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 0x7f8cac15d2b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400e9b0 tx=0x7f8c9400ee70 comp rx=0 tx=0).stop 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 shutdown_connections 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cac1437d0 0x7f8cac15c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0c9410 0x7f8cac15d2b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 --2- 172.21.15.157:0/915770166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cac070330 0x7f8cac1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.227+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 >> 172.21.15.157:0/915770166 conn(0x7f8cac0718f0 msgr2=0x7f8cac0cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:01.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.228+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 shutdown_connections 2024-03-23T15:47:01.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.228+0000 7f8cb21a5640 1 -- 172.21.15.157:0/915770166 wait complete. 2024-03-23T15:47:01.241 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new f4438f7e-f95e-4ae6-91aa-4cedeb8b31db 6 2024-03-23T15:47:01.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.341+0000 7f5813cb5640 1 Processor -- start 2024-03-23T15:47:01.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.341+0000 7f5813cb5640 1 -- start start 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.342+0000 7f5813cb5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f580c152310 0x7f580c0c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.342+0000 7f5813cb5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f580c0c98a0 0x7f580c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.342+0000 7f5813cb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c0c3e40 0x7f580c0cd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.342+0000 7f5813cb5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f580c0c9cc0 con 0x7f580c152310 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.342+0000 7f5813cb5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f580c158090 con 0x7f580c0c98a0 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.342+0000 7f5813cb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f580c158330 con 0x7f580c0c3e40 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.343+0000 7f5811a51640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f580c152310 0x7f580c0c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.343+0000 7f5812252640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c0c3e40 0x7f580c0cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.346 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.343+0000 7f5811250640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f580c0c98a0 0x7f580c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.347 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.343+0000 7f5811a51640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f580c152310 0x7f580c0c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:48036/0 (socket says 172.21.15.157:48036) 2024-03-23T15:47:01.347 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.343+0000 7f5811a51640 1 -- 172.21.15.157:0/418696402 learned_addr learned my addr 172.21.15.157:0/418696402 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:01.348 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.345+0000 7f5811a51640 1 -- 172.21.15.157:0/418696402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c0c3e40 msgr2=0x7f580c0cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:01.348 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.345+0000 7f5811a51640 1 --2- 172.21.15.157:0/418696402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c0c3e40 0x7f580c0cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.348 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.345+0000 7f5811a51640 1 -- 172.21.15.157:0/418696402 >> v2:172.21.15.157:3300/0 conn(0x7f580c0c98a0 msgr2=0x7f580c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:01.348 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.345+0000 7f5811a51640 1 --2- 172.21.15.157:0/418696402 >> v2:172.21.15.157:3300/0 conn(0x7f580c0c98a0 0x7f580c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.349 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.345+0000 7f5811a51640 1 -- 172.21.15.157:0/418696402 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c0c33f0 con 0x7f580c152310 2024-03-23T15:47:01.350 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.347+0000 7f5811a51640 1 --2- 172.21.15.157:0/418696402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c152310 0x7f580c0c9360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f580c073930 tx=0x7f57fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4190f74da2756ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:01.353 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5810a4f640 1 -- 172.21.15.157:0/418696402 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc031870 con 0x7f580c152310 2024-03-23T15:47:01.354 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5810a4f640 1 -- 172.21.15.157:0/418696402 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc0319d0 con 0x7f580c152310 2024-03-23T15:47:01.354 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5810a4f640 1 -- 172.21.15.157:0/418696402 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc031d60 con 0x7f580c152310 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 -- 172.21.15.157:0/418696402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c152310 msgr2=0x7f580c0c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 --2- 172.21.15.157:0/418696402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c152310 0x7f580c0c9360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f580c073930 tx=0x7f57fc02b910 comp rx=0 tx=0).stop 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 -- 172.21.15.157:0/418696402 shutdown_connections 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 --2- 172.21.15.157:0/418696402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c0c3e40 0x7f580c0cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 --2- 172.21.15.157:0/418696402 >> v2:172.21.15.157:3300/0 conn(0x7f580c0c98a0 0x7f580c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 --2- 172.21.15.157:0/418696402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c152310 0x7f580c0c9360 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 -- 172.21.15.157:0/418696402 >> 172.21.15.157:0/418696402 conn(0x7f580c061b50 msgr2=0x7f580c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 -- 172.21.15.157:0/418696402 shutdown_connections 2024-03-23T15:47:01.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.351+0000 7f5813cb5640 1 -- 172.21.15.157:0/418696402 wait complete. 2024-03-23T15:47:01.356 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 Processor -- start 2024-03-23T15:47:01.356 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 -- start start 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 0x7f580c0ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f580c0c98a0 0x7f580c14c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c152310 0x7f580c14cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f580c0b2080 con 0x7f580c0c3e40 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f580c1553d0 con 0x7f580c0c98a0 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.353+0000 7f5813cb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f580c154af0 con 0x7f580c152310 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.354+0000 7f5811a51640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 0x7f580c0ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.354+0000 7f5811250640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f580c0c98a0 0x7f580c14c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.354+0000 7f5812252640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c152310 0x7f580c14cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.357 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.354+0000 7f5811a51640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 0x7f580c0ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:48038/0 (socket says 172.21.15.157:48038) 2024-03-23T15:47:01.358 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.354+0000 7f5811a51640 1 -- 172.21.15.157:0/432673346 learned_addr learned my addr 172.21.15.157:0/432673346 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:01.360 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.357+0000 7f5811a51640 1 -- 172.21.15.157:0/432673346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c152310 msgr2=0x7f580c14cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.361 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.357+0000 7f5811a51640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c152310 0x7f580c14cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.361 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.357+0000 7f5811a51640 1 -- 172.21.15.157:0/432673346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f580c0c98a0 msgr2=0x7f580c14c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.361 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.357+0000 7f5811a51640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f580c0c98a0 0x7f580c14c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.361 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.357+0000 7f5811a51640 1 -- 172.21.15.157:0/432673346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc0094d0 con 0x7f580c0c3e40 2024-03-23T15:47:01.362 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.360+0000 7f5811a51640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 0x7f580c0ccba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f57fc02b950 tx=0x7f57fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:01.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.363+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc03ac60 con 0x7f580c0c3e40 2024-03-23T15:47:01.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.363+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc03adc0 con 0x7f580c0c3e40 2024-03-23T15:47:01.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.363+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc044630 con 0x7f580c0c3e40 2024-03-23T15:47:01.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.363+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580c155900 con 0x7f580c0c3e40 2024-03-23T15:47:01.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.363+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580c0b1a30 con 0x7f580c0c3e40 2024-03-23T15:47:01.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.364+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f580c15a040 con 0x7f580c0c3e40 2024-03-23T15:47:01.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.365+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f57fc044790 con 0x7f580c0c3e40 2024-03-23T15:47:01.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.365+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f57fc02d040 con 0x7f580c0c3e40 2024-03-23T15:47:01.469 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.466+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f57fc052060 con 0x7f580c0c3e40 2024-03-23T15:47:01.551 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:01.548+0000 7fbb1d63e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T15:47:01.565 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.562+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f4438f7e-f95e-4ae6-91aa-4cedeb8b31db", "id": 6} v 0) -- 0x7f580c0b0ad0 con 0x7f580c0c3e40 2024-03-23T15:47:01.611 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.608+0000 7f58027fc640 1 -- 172.21.15.157:0/432673346 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4438f7e-f95e-4ae6-91aa-4cedeb8b31db", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f57fc035020 con 0x7f580c0c3e40 2024-03-23T15:47:01.612 INFO:teuthology.orchestra.run.smithi157.stdout:6 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 msgr2=0x7f580c0ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 0x7f580c0ccba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f57fc02b950 tx=0x7f57fc0046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 shutdown_connections 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f580c152310 0x7f580c14cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f580c0c98a0 0x7f580c14c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 --2- 172.21.15.157:0/432673346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0c3e40 0x7f580c0ccba0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 >> 172.21.15.157:0/432673346 conn(0x7f580c061b50 msgr2=0x7f580c150c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 shutdown_connections 2024-03-23T15:47:01.613 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.609+0000 7f5813cb5640 1 -- 172.21.15.157:0/432673346 wait complete. 2024-03-23T15:47:01.624 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 21ca1071-c633-4690-a97b-15ff27ea72b7 7 2024-03-23T15:47:01.727 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.724+0000 7f2cc5f3a640 1 Processor -- start 2024-03-23T15:47:01.728 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.724+0000 7f2cc5f3a640 1 -- start start 2024-03-23T15:47:01.728 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cc5f3a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2cc0151320 0x7f2cc0157a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cc5f3a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc0157fa0 con 0x7f2cc00c6f70 2024-03-23T15:47:01.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cc5f3a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc0151700 con 0x7f2cc015a900 2024-03-23T15:47:01.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cc5f3a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc0070060 con 0x7f2cc0151320 2024-03-23T15:47:01.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cb7fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f2cc00c6f70 0x7f2cc00c5800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.157:37148/0 (socket says 172.21.15.157:37148) 2024-03-23T15:47:01.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cb7fff640 1 -- 172.21.15.157:0/1455166121 learned_addr learned my addr 172.21.15.157:0/1455166121 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-23T15:47:01.729 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.725+0000 7f2cbf7fe640 1 --2- 172.21.15.157:0/1455166121 >> v2:172.21.15.157:3300/0 conn(0x7f2cc0151320 0x7f2cc0157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.730 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.727+0000 7f2cbeffd640 1 -- 172.21.15.157:0/1455166121 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1529141000 0 0) 0x7f2cc0157fa0 con 0x7f2cc00c6f70 2024-03-23T15:47:01.731 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.728+0000 7f2cbeffd640 1 -- 172.21.15.157:0/1455166121 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cac003520 con 0x7f2cc00c6f70 2024-03-23T15:47:01.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.729+0000 7f2cbf7fe640 1 -- 172.21.15.157:0/1455166121 >> v1:172.21.15.157:6789/0 conn(0x7f2cc015a900 legacy=0x7f2cc015ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:01.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.729+0000 7f2cbf7fe640 1 -- 172.21.15.157:0/1455166121 >> v1:172.21.15.129:6789/0 conn(0x7f2cc00c6f70 legacy=0x7f2cc00c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.732 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.729+0000 7f2cbf7fe640 1 -- 172.21.15.157:0/1455166121 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc0070340 con 0x7f2cc0151320 2024-03-23T15:47:01.735 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.732+0000 7f2cbf7fe640 1 --2- 172.21.15.157:0/1455166121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc0151320 0x7f2cc0157a60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00097a0 tx=0x7f2cb002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4073cf48e48d4d2a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:01.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.738+0000 7f2cbeffd640 1 -- 172.21.15.157:0/1455166121 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0031830 con 0x7f2cc0151320 2024-03-23T15:47:01.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.738+0000 7f2cbeffd640 1 -- 172.21.15.157:0/1455166121 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb0031990 con 0x7f2cc0151320 2024-03-23T15:47:01.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.738+0000 7f2cbeffd640 1 -- 172.21.15.157:0/1455166121 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0031d20 con 0x7f2cc0151320 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.738+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/1455166121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc0151320 msgr2=0x7f2cc0157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.738+0000 7f2cc5f3a640 1 --2- 172.21.15.157:0/1455166121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc0151320 0x7f2cc0157a60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00097a0 tx=0x7f2cb002b8d0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.739+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/1455166121 shutdown_connections 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.739+0000 7f2cc5f3a640 1 --2- 172.21.15.157:0/1455166121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc0151320 0x7f2cc0157a60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.739+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/1455166121 >> 172.21.15.157:0/1455166121 conn(0x7f2cc006c240 msgr2=0x7f2cc006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.739+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/1455166121 shutdown_connections 2024-03-23T15:47:01.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.739+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/1455166121 wait complete. 2024-03-23T15:47:01.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 Processor -- start 2024-03-23T15:47:01.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 -- start start 2024-03-23T15:47:01.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc00c6f70 0x7f2cc00c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cc0151320 0x7f2cc0155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2cc015a900 0x7f2cc0156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:01.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc00c8f20 con 0x7f2cc0151320 2024-03-23T15:47:01.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc00c7db0 con 0x7f2cc00c6f70 2024-03-23T15:47:01.744 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.740+0000 7f2cc5f3a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc00c1630 con 0x7f2cc015a900 2024-03-23T15:47:01.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.741+0000 7f2cbf7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc00c6f70 0x7f2cc00c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.741+0000 7f2cbffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2cc015a900 0x7f2cc0156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.741+0000 7f2cb7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cc0151320 0x7f2cc0155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:01.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.741+0000 7f2cbffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2cc015a900 0x7f2cc0156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:55456/0 (socket says 172.21.15.157:55456) 2024-03-23T15:47:01.745 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.741+0000 7f2cbffff640 1 -- 172.21.15.157:0/797622401 learned_addr learned my addr 172.21.15.157:0/797622401 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-23T15:47:01.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.744+0000 7f2cb7fff640 1 -- 172.21.15.157:0/797622401 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2cc015a900 msgr2=0x7f2cc0156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.744+0000 7f2cb7fff640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2cc015a900 0x7f2cc0156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.744+0000 7f2cb7fff640 1 -- 172.21.15.157:0/797622401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc00c6f70 msgr2=0x7f2cc00c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.744+0000 7f2cb7fff640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc00c6f70 0x7f2cc00c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.744+0000 7f2cb7fff640 1 -- 172.21.15.157:0/797622401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb0009400 con 0x7f2cc0151320 2024-03-23T15:47:01.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.747+0000 7f2cb7fff640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cc0151320 0x7f2cc0155fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c3c0 tx=0x7f2ca8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:01.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.750+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8005ce0 con 0x7f2cc0151320 2024-03-23T15:47:01.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.750+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca8005e40 con 0x7f2cc0151320 2024-03-23T15:47:01.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.750+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8007310 con 0x7f2cc0151320 2024-03-23T15:47:01.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.750+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc00c82e0 con 0x7f2cc0151320 2024-03-23T15:47:01.753 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.750+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc00c86f0 con 0x7f2cc0151320 2024-03-23T15:47:01.754 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.751+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc00ca030 con 0x7f2cc0151320 2024-03-23T15:47:01.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.752+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2ca8007470 con 0x7f2cc0151320 2024-03-23T15:47:01.849 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:01.846+0000 7fbb1d63e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T15:47:01.849 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.846+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f2ca8007720 con 0x7f2cc0151320 2024-03-23T15:47:01.853 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.850+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2ca8017070 con 0x7f2cc0151320 2024-03-23T15:47:01.901 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:01.898+0000 7fbb1d63e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T15:47:01.923 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.920+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "21ca1071-c633-4690-a97b-15ff27ea72b7", "id": 7} v 0) -- 0x7f2cc0074b20 con 0x7f2cc0151320 2024-03-23T15:47:01.974 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.971+0000 7f2cbcff9640 1 -- 172.21.15.157:0/797622401 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21ca1071-c633-4690-a97b-15ff27ea72b7", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f2ca801c630 con 0x7f2cc0151320 2024-03-23T15:47:01.975 INFO:teuthology.orchestra.run.smithi157.stdout:7 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cc0151320 msgr2=0x7f2cc0155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cc0151320 0x7f2cc0155fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c3c0 tx=0x7f2ca8001e70 comp rx=0 tx=0).stop 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 shutdown_connections 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2cc015a900 0x7f2cc0156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cc0151320 0x7f2cc0155fd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 --2- 172.21.15.157:0/797622401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2cc00c6f70 0x7f2cc00c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 >> 172.21.15.157:0/797622401 conn(0x7f2cc006c240 msgr2=0x7f2cc014b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 shutdown_connections 2024-03-23T15:47:01.976 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-23T15:47:01.972+0000 7f2cc5f3a640 1 -- 172.21.15.157:0/797622401 wait complete. 2024-03-23T15:47:01.986 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-23T15:47:01.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-03-23T15:47:01.989 INFO:tasks.ceph.osd.0:Started 2024-03-23T15:47:01.989 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-23T15:47:01.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-03-23T15:47:01.993 INFO:tasks.ceph.osd.1:Started 2024-03-23T15:47:01.993 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-23T15:47:01.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-03-23T15:47:01.995 INFO:tasks.ceph.osd.2:Started 2024-03-23T15:47:01.996 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-23T15:47:01.996 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-03-23T15:47:01.998 INFO:tasks.ceph.osd.3:Started 2024-03-23T15:47:01.998 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-23T15:47:01.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-03-23T15:47:02.000 INFO:tasks.ceph.osd.4:Started 2024-03-23T15:47:02.000 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-23T15:47:02.000 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-03-23T15:47:02.002 INFO:tasks.ceph.osd.5:Started 2024-03-23T15:47:02.002 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-23T15:47:02.002 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-03-23T15:47:02.004 INFO:tasks.ceph.osd.6:Started 2024-03-23T15:47:02.004 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-23T15:47:02.005 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-03-23T15:47:02.007 INFO:tasks.ceph.osd.7:Started 2024-03-23T15:47:02.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:02.055 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:02.052+0000 7fbb1d63e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T15:47:02.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.127+0000 7fa4c2247640 1 Processor -- start 2024-03-23T15:47:02.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.127+0000 7fa4c2247640 1 -- start start 2024-03-23T15:47:02.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.128+0000 7fa4c2247640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa4bc06f900 0x7fa4bc0c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:02.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.128+0000 7fa4c2247640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 0x7fa4bc0c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:02.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.128+0000 7fa4c2247640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc0c41a0 con 0x7fa4bc152440 2024-03-23T15:47:02.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.128+0000 7fa4c2247640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa4bc0c99f0 con 0x7fa4bc06f900 2024-03-23T15:47:02.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.128+0000 7fa4c2247640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa4bc06ed60 con 0x7fa4bc0ce4e0 2024-03-23T15:47:02.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.129+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 0x7fa4bc0c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:02.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.129+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 0x7fa4bc0c37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:47036/0 (socket says 172.21.15.129:47036) 2024-03-23T15:47:02.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.129+0000 7fa4bbfff640 1 -- 172.21.15.129:0/3276698632 learned_addr learned my addr 172.21.15.129:0/3276698632 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:02.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.131+0000 7fa4bbfff640 1 -- 172.21.15.129:0/3276698632 >> v2:172.21.15.157:3300/0 conn(0x7fa4bc06f900 msgr2=0x7fa4bc0c3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:02.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.131+0000 7fa4bbfff640 1 --2- 172.21.15.129:0/3276698632 >> v2:172.21.15.157:3300/0 conn(0x7fa4bc06f900 0x7fa4bc0c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.131+0000 7fa4bbfff640 1 -- 172.21.15.129:0/3276698632 >> v1:172.21.15.157:6789/0 conn(0x7fa4bc152440 legacy=0x7fa4bc06f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:02.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.131+0000 7fa4bbfff640 1 -- 172.21.15.129:0/3276698632 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4bc15c670 con 0x7fa4bc0ce4e0 2024-03-23T15:47:02.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.133+0000 7fa4bbfff640 1 --2- 172.21.15.129:0/3276698632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 0x7fa4bc0c37d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009e50 tx=0x7fa4b402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e0e10b6829ff163 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:02.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.138+0000 7fa4ba7fc640 1 -- 172.21.15.129:0/3276698632 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b402cb00 con 0x7fa4bc0ce4e0 2024-03-23T15:47:02.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.138+0000 7fa4ba7fc640 1 -- 172.21.15.129:0/3276698632 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b402cc60 con 0x7fa4bc0ce4e0 2024-03-23T15:47:02.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.138+0000 7fa4ba7fc640 1 -- 172.21.15.129:0/3276698632 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b40343f0 con 0x7fa4bc0ce4e0 2024-03-23T15:47:02.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 -- 172.21.15.129:0/3276698632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 msgr2=0x7fa4bc0c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 --2- 172.21.15.129:0/3276698632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 0x7fa4bc0c37d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009e50 tx=0x7fa4b402bf60 comp rx=0 tx=0).stop 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 -- 172.21.15.129:0/3276698632 shutdown_connections 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 --2- 172.21.15.129:0/3276698632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc0ce4e0 0x7fa4bc0c37d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 --2- 172.21.15.129:0/3276698632 >> v2:172.21.15.157:3300/0 conn(0x7fa4bc06f900 0x7fa4bc0c3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 -- 172.21.15.129:0/3276698632 >> 172.21.15.129:0/3276698632 conn(0x7fa4bc0cd8a0 msgr2=0x7fa4bc0cdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 -- 172.21.15.129:0/3276698632 shutdown_connections 2024-03-23T15:47:02.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.139+0000 7fa4c2247640 1 -- 172.21.15.129:0/3276698632 wait complete. 2024-03-23T15:47:02.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.140+0000 7fa4c2247640 1 Processor -- start 2024-03-23T15:47:02.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.140+0000 7fa4c2247640 1 -- start start 2024-03-23T15:47:02.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.141+0000 7fa4c2247640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 0x7fa4bc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4c2247640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4bc0ce4e0 0x7fa4bc156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4c2247640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4bc152440 0x7fa4bc1f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4c2247640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc157460 con 0x7fa4bc0ce4e0 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4c2247640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc06e140 con 0x7fa4bc152440 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4c2247640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa4bc158420 con 0x7fa4bc06f900 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4bb7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 0x7fa4bc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:02.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4bb7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 0x7fa4bc0cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:47052/0 (socket says 172.21.15.129:47052) 2024-03-23T15:47:02.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.142+0000 7fa4bb7fe640 1 -- 172.21.15.129:0/2222114464 learned_addr learned my addr 172.21.15.129:0/2222114464 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:02.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.145+0000 7fa4bb7fe640 1 -- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4bc152440 msgr2=0x7fa4bc1f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:02.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.145+0000 7fa4bb7fe640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4bc152440 0x7fa4bc1f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.145+0000 7fa4bb7fe640 1 -- 172.21.15.129:0/2222114464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4bc0ce4e0 msgr2=0x7fa4bc156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:02.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.145+0000 7fa4bb7fe640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4bc0ce4e0 0x7fa4bc156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.145+0000 7fa4bb7fe640 1 -- 172.21.15.129:0/2222114464 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b4009b30 con 0x7fa4bc06f900 2024-03-23T15:47:02.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.148+0000 7fa4bb7fe640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 0x7fa4bc0cc990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00b410 tx=0x7fa4ac00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:02.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.152+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac011830 con 0x7fa4bc06f900 2024-03-23T15:47:02.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.152+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ac011990 con 0x7fa4bc06f900 2024-03-23T15:47:02.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.152+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac011d40 con 0x7fa4bc06f900 2024-03-23T15:47:02.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.152+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4bc157a50 con 0x7fa4bc06f900 2024-03-23T15:47:02.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.152+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4bc06da90 con 0x7fa4bc06f900 2024-03-23T15:47:02.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.153+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4bc070180 con 0x7fa4bc06f900 2024-03-23T15:47:02.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.153+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4ac0247d0 con 0x7fa4bc06f900 2024-03-23T15:47:02.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.251+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa4ac01a400 con 0x7fa4bc06f900 2024-03-23T15:47:02.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.251+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa4ac039dd0 con 0x7fa4bc06f900 2024-03-23T15:47:02.290 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:02.287+0000 7f8cae642040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T15:47:02.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.314+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4bc06ddd0 con 0x7fa4bc06f900 2024-03-23T15:47:02.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.517+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mgrmap(e 2) ==== 44915+0+0 (secure 0 0 0) 0x7fa4ac0233e0 con 0x7fa4bc06f900 2024-03-23T15:47:02.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.568+0000 7fa4a3fff640 1 -- 172.21.15.129:0/2222114464 <== mon.2 v2:172.21.15.157:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa4ac00f2e0 con 0x7fa4bc06f900 2024-03-23T15:47:02.571 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:02.571 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":10,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:01.927700+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 msgr2=0x7fa4bc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 0x7fa4bc0cc990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00b410 tx=0x7fa4ac00b8d0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 shutdown_connections 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4bc152440 0x7fa4bc1f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4bc0ce4e0 0x7fa4bc156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 --2- 172.21.15.129:0/2222114464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa4bc06f900 0x7fa4bc0cc990 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:02.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 >> 172.21.15.129:0/2222114464 conn(0x7fa4bc0cd8a0 msgr2=0x7fa4bc151480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:02.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 shutdown_connections 2024-03-23T15:47:02.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:02.569+0000 7fa4c2247640 1 -- 172.21.15.129:0/2222114464 wait complete. 2024-03-23T15:47:02.583 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-23T15:47:02.583 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-23T15:47:02.600 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:02.597+0000 7f8cae642040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T15:47:02.615 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-23T15:47:02.613+0000 7fc2b52a4640 -1 Falling back to public interface 2024-03-23T15:47:02.631 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-23T15:47:02.629+0000 7fdd65224640 -1 Falling back to public interface 2024-03-23T15:47:02.643 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-23T15:47:02.641+0000 7f68189e1640 -1 Falling back to public interface 2024-03-23T15:47:02.644 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-03-23T15:47:02.641+0000 7fb898ec8640 -1 Falling back to public interface 2024-03-23T15:47:02.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T15:47:02.641+0000 7fbcefb8f640 -1 Falling back to public interface 2024-03-23T15:47:02.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T15:47:02.641+0000 7fbceeb3e640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-03-23T15:47:02.644 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T15:47:02.641+0000 7fbceeb3e640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-03-23T15:47:02.652 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-03-23T15:47:02.649+0000 7f8bf5e0f640 -1 Falling back to public interface 2024-03-23T15:47:02.657 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:02.653+0000 7f8cae642040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T15:47:02.664 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-03-23T15:47:02.661+0000 7fc868215640 -1 Falling back to public interface 2024-03-23T15:47:02.677 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-23T15:47:02.675+0000 7fc780268640 -1 Falling back to public interface 2024-03-23T15:47:02.794 INFO:tasks.ceph.mgr.y.smithi129.stderr:2024-03-23T15:47:02.792+0000 7f8cae642040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T15:47:03.389 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:03.479 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.476+0000 7f136d31e640 1 Processor -- start 2024-03-23T15:47:03.479 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.476+0000 7f136d31e640 1 -- start start 2024-03-23T15:47:03.480 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.477+0000 7f136d31e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13681546b0 0x7f136806f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:03.481 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.477+0000 7f136d31e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136806ff90 con 0x7f136806e2b0 2024-03-23T15:47:03.481 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.477+0000 7f136d31e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1368070830 con 0x7f1368070cc0 2024-03-23T15:47:03.481 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.477+0000 7f136d31e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f136815aef0 con 0x7f13681546b0 2024-03-23T15:47:03.481 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.478+0000 7f13667fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f136806e2b0 0x7f13680c7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:39466/0 (socket says 172.21.15.129:39466) 2024-03-23T15:47:03.481 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.478+0000 7f13667fc640 1 -- 172.21.15.129:0/75496364 learned_addr learned my addr 172.21.15.129:0/75496364 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:03.481 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.478+0000 7f1366ffd640 1 --2- 172.21.15.129:0/75496364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13681546b0 0x7f136806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:03.483 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.480+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1287900551 0 0) 0x7f136806ff90 con 0x7f136806e2b0 2024-03-23T15:47:03.483 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.480+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1350003520 con 0x7f136806e2b0 2024-03-23T15:47:03.484 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.480+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626922493 0 0) 0x7f1368070830 con 0x7f1368070cc0 2024-03-23T15:47:03.484 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.481+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f136806ff90 con 0x7f1368070cc0 2024-03-23T15:47:03.484 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.481+0000 7f1366ffd640 1 -- 172.21.15.129:0/75496364 >> v1:172.21.15.157:6789/0 conn(0x7f1368070cc0 legacy=0x7f136806b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:03.484 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.481+0000 7f1366ffd640 1 -- 172.21.15.129:0/75496364 >> v1:172.21.15.129:6789/0 conn(0x7f136806e2b0 legacy=0x7f13680c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:03.484 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.481+0000 7f1366ffd640 1 -- 172.21.15.129:0/75496364 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136815b1b0 con 0x7f13681546b0 2024-03-23T15:47:03.485 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.482+0000 7f1366ffd640 1 --2- 172.21.15.129:0/75496364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13681546b0 0x7f136806f120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f135c0097d0 tx=0x7f135c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=578280e593b01333 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:03.489 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.486+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c030880 con 0x7f13681546b0 2024-03-23T15:47:03.489 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.486+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f135c0309e0 con 0x7f13681546b0 2024-03-23T15:47:03.489 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.486+0000 7f1365ffb640 1 -- 172.21.15.129:0/75496364 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c030d70 con 0x7f13681546b0 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 -- 172.21.15.129:0/75496364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13681546b0 msgr2=0x7f136806f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 --2- 172.21.15.129:0/75496364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13681546b0 0x7f136806f120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f135c0097d0 tx=0x7f135c02b9c0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 -- 172.21.15.129:0/75496364 shutdown_connections 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 --2- 172.21.15.129:0/75496364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13681546b0 0x7f136806f120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 -- 172.21.15.129:0/75496364 >> 172.21.15.129:0/75496364 conn(0x7f13680c9340 msgr2=0x7f13680c9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 -- 172.21.15.129:0/75496364 shutdown_connections 2024-03-23T15:47:03.490 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.487+0000 7f136d31e640 1 -- 172.21.15.129:0/75496364 wait complete. 2024-03-23T15:47:03.491 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.488+0000 7f136d31e640 1 Processor -- start 2024-03-23T15:47:03.491 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.488+0000 7f136d31e640 1 -- start start 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f136d31e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f136806e2b0 0x7f136814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f136d31e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070cc0 0x7f1368155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f136d31e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 0x7f13681560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f136d31e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1368155750 con 0x7f13681546b0 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f136d31e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1368159c00 con 0x7f1368070cc0 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f136d31e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f136814c770 con 0x7f136806e2b0 2024-03-23T15:47:03.492 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.489+0000 7f13677fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 0x7f13681560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:03.493 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.490+0000 7f13667fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070cc0 0x7f1368155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:03.493 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.490+0000 7f1366ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f136806e2b0 0x7f136814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:03.493 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.490+0000 7f13677fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 0x7f13681560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44992/0 (socket says 172.21.15.129:44992) 2024-03-23T15:47:03.493 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.490+0000 7f13677fe640 1 -- 172.21.15.129:0/1383678721 learned_addr learned my addr 172.21.15.129:0/1383678721 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:03.495 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.493+0000 7f13677fe640 1 -- 172.21.15.129:0/1383678721 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f136806e2b0 msgr2=0x7f136814bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:03.496 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.493+0000 7f13677fe640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f136806e2b0 0x7f136814bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.496 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.493+0000 7f13677fe640 1 -- 172.21.15.129:0/1383678721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070cc0 msgr2=0x7f1368155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:03.496 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.493+0000 7f13677fe640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070cc0 0x7f1368155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.496 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.493+0000 7f13677fe640 1 -- 172.21.15.129:0/1383678721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135c009400 con 0x7f13681546b0 2024-03-23T15:47:03.497 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.495+0000 7f13677fe640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 0x7f13681560e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f136000c790 tx=0x7f136000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:03.501 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.498+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136000cea0 con 0x7f13681546b0 2024-03-23T15:47:03.501 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.498+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1360003b70 con 0x7f13681546b0 2024-03-23T15:47:03.501 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.498+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136000a850 con 0x7f13681546b0 2024-03-23T15:47:03.501 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.498+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1368157700 con 0x7f13681546b0 2024-03-23T15:47:03.501 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.498+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1368159730 con 0x7f13681546b0 2024-03-23T15:47:03.502 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.499+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13681558c0 con 0x7f13681546b0 2024-03-23T15:47:03.616 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.614+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 2) ==== 44915+0+0 (secure 0 0 0) 0x7f136000a9b0 con 0x7f13681546b0 2024-03-23T15:47:03.630 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.627+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1360011380 con 0x7f13681546b0 2024-03-23T15:47:03.630 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.628+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1360012a70 con 0x7f13681546b0 2024-03-23T15:47:03.692 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.690+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1368157700 con 0x7f13681546b0 2024-03-23T15:47:03.816 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.813+0000 7f134f7fe640 1 -- 172.21.15.129:0/1383678721 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1368157700 con 0x7f13681546b0 2024-03-23T15:47:03.817 INFO:teuthology.misc.health.smithi129.stdout: 2024-03-23T15:47:03.817 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:01.927700+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 msgr2=0x7f13681560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 0x7f13681560e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f136000c790 tx=0x7f136000cc50 comp rx=0 tx=0).stop 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 shutdown_connections 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13681546b0 0x7f13681560e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070cc0 0x7f1368155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 --2- 172.21.15.129:0/1383678721 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f136806e2b0 0x7f136814bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 >> 172.21.15.129:0/1383678721 conn(0x7f13680c9340 msgr2=0x7f13680ca640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 shutdown_connections 2024-03-23T15:47:03.818 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:03.814+0000 7f136d31e640 1 -- 172.21.15.129:0/1383678721 wait complete. 2024-03-23T15:47:03.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-23T15:47:04.257 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:04.255+0000 7fbabe2dc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-23T15:47:04.276 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T15:47:04.274+0000 7fbabe2dc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-23T15:47:05.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:47:05.640 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:47:06.066 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-23T15:47:06.064+0000 7fc2b52a4640 -1 osd.5 0 log_to_monitors true 2024-03-23T15:47:06.107 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-03-23T15:47:06.105+0000 7fb898ec8640 -1 osd.2 0 log_to_monitors true 2024-03-23T15:47:06.114 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-23T15:47:06.112+0000 7fdd65224640 -1 osd.6 0 log_to_monitors true 2024-03-23T15:47:06.150 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-23T15:47:06.148+0000 7f68189e1640 -1 osd.7 0 log_to_monitors true 2024-03-23T15:47:06.166 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-03-23T15:47:06.163+0000 7f8bf5e0f640 -1 osd.1 0 log_to_monitors true 2024-03-23T15:47:06.166 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T15:47:06.163+0000 7fbcefb8f640 -1 osd.3 0 log_to_monitors true 2024-03-23T15:47:06.188 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-23T15:47:06.186+0000 7fc780268640 -1 osd.4 0 log_to_monitors true 2024-03-23T15:47:06.201 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-03-23T15:47:06.198+0000 7fc868215640 -1 osd.0 0 log_to_monitors true 2024-03-23T15:47:09.195 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-23T15:47:09.192+0000 7f6814977640 -1 osd.7 0 waiting for initial osdmap 2024-03-23T15:47:09.222 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-03-23T15:47:09.220+0000 7fc863bb7640 -1 osd.0 0 waiting for initial osdmap 2024-03-23T15:47:09.232 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-23T15:47:09.230+0000 7fc77c1fe640 -1 osd.4 0 waiting for initial osdmap 2024-03-23T15:47:09.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T15:47:09.231+0000 7fbcebb25640 -1 osd.3 0 waiting for initial osdmap 2024-03-23T15:47:09.252 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-23T15:47:09.250+0000 7fdd611ba640 -1 osd.6 0 waiting for initial osdmap 2024-03-23T15:47:09.266 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-03-23T15:47:09.263+0000 7fb895672640 -1 osd.2 0 waiting for initial osdmap 2024-03-23T15:47:09.266 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-03-23T15:47:09.263+0000 7f8bf1da5640 -1 osd.1 0 waiting for initial osdmap 2024-03-23T15:47:09.315 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-23T15:47:09.313+0000 7fc2b1a4e640 -1 osd.5 0 waiting for initial osdmap 2024-03-23T15:47:09.322 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-03-23T15:47:09.320+0000 7f8be9bc5640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.327 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T15:47:09.325+0000 7fbce3945640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.332 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-03-23T15:47:09.330+0000 7fc85b9d7640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.333 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-03-23T15:47:09.330+0000 7fb88cc7e640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.342 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-23T15:47:09.340+0000 7fc77401e640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.346 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-23T15:47:09.344+0000 7fdd58fda640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.346 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-23T15:47:09.344+0000 7f680c797640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:09.373 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-23T15:47:09.371+0000 7fc2a905a640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T15:47:10.633 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:10.723 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.720+0000 7f7b8138a640 1 Processor -- start 2024-03-23T15:47:10.723 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.721+0000 7f7b8138a640 1 -- start start 2024-03-23T15:47:10.724 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.721+0000 7f7b8138a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7b7c15d920 0x7f7b7c15ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:10.724 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.721+0000 7f7b8138a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7c0cf730 con 0x7f7b7c15d0e0 2024-03-23T15:47:10.724 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.721+0000 7f7b8138a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7c0ceb90 con 0x7f7b7c15d4c0 2024-03-23T15:47:10.724 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.721+0000 7f7b8138a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f7b7c0cdca0 con 0x7f7b7c15d920 2024-03-23T15:47:10.725 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.722+0000 7f7b7b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7b7c15d920 0x7f7b7c15ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:10.725 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.722+0000 7f7b7affd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7b7c15d0e0 0x7f7b7c0cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38578/0 (socket says 172.21.15.129:38578) 2024-03-23T15:47:10.725 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.722+0000 7f7b7affd640 1 -- 172.21.15.129:0/943685084 learned_addr learned my addr 172.21.15.129:0/943685084 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:10.728 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.725+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/943685084 >> v1:172.21.15.157:6789/0 conn(0x7f7b7c15d4c0 legacy=0x7f7b7c06c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:10.728 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.725+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/943685084 >> v1:172.21.15.129:6789/0 conn(0x7f7b7c15d0e0 legacy=0x7f7b7c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:10.728 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.725+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/943685084 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c0cdf80 con 0x7f7b7c15d920 2024-03-23T15:47:10.732 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.730+0000 7f7b7b7fe640 1 --2- 172.21.15.129:0/943685084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c15d920 0x7f7b7c15ddd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009d90 tx=0x7f7b7402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c36e3aa1e8c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:10.736 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.733+0000 7f7b79ffb640 1 -- 172.21.15.129:0/943685084 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7402cd00 con 0x7f7b7c15d920 2024-03-23T15:47:10.736 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.733+0000 7f7b79ffb640 1 -- 172.21.15.129:0/943685084 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7402ce60 con 0x7f7b7c15d920 2024-03-23T15:47:10.736 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.733+0000 7f7b79ffb640 1 -- 172.21.15.129:0/943685084 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74034660 con 0x7f7b7c15d920 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 -- 172.21.15.129:0/943685084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c15d920 msgr2=0x7f7b7c15ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 --2- 172.21.15.129:0/943685084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c15d920 0x7f7b7c15ddd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009d90 tx=0x7f7b7402bf50 comp rx=0 tx=0).stop 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 -- 172.21.15.129:0/943685084 shutdown_connections 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 --2- 172.21.15.129:0/943685084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c15d920 0x7f7b7c15ddd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 -- 172.21.15.129:0/943685084 >> 172.21.15.129:0/943685084 conn(0x7f7b7c0cfb40 msgr2=0x7f7b7c0cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 -- 172.21.15.129:0/943685084 shutdown_connections 2024-03-23T15:47:10.737 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.734+0000 7f7b8138a640 1 -- 172.21.15.129:0/943685084 wait complete. 2024-03-23T15:47:10.738 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.736+0000 7f7b8138a640 1 Processor -- start 2024-03-23T15:47:10.738 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.736+0000 7f7b8138a640 1 -- start start 2024-03-23T15:47:10.739 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.736+0000 7f7b8138a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b7c15d0e0 0x7f7b7c1f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:10.739 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.736+0000 7f7b8138a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b7c15d920 0x7f7b7c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:10.739 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.736+0000 7f7b8138a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 0x7f7b7c0b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:10.740 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.737+0000 7f7b8138a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0b2cc0 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.740 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.737+0000 7f7b8138a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0b5520 con 0x7f7b7c15d920 2024-03-23T15:47:10.740 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.737+0000 7f7b8138a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c14ea80 con 0x7f7b7c15d0e0 2024-03-23T15:47:10.740 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.737+0000 7f7b7b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 0x7f7b7c0b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:10.740 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.737+0000 7f7b7b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 0x7f7b7c0b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59718/0 (socket says 172.21.15.129:59718) 2024-03-23T15:47:10.740 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.737+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/3462575070 learned_addr learned my addr 172.21.15.129:0/3462575070 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:10.743 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.740+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b7c15d0e0 msgr2=0x7f7b7c1f0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:10.743 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.740+0000 7f7b7b7fe640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b7c15d0e0 0x7f7b7c1f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:10.743 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.740+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b7c15d920 msgr2=0x7f7b7c151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:10.743 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.740+0000 7f7b7b7fe640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b7c15d920 0x7f7b7c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:10.743 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.740+0000 7f7b7b7fe640 1 -- 172.21.15.129:0/3462575070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b74009a70 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.745 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.743+0000 7f7b7b7fe640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 0x7f7b7c0b30e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009eb0 tx=0x7f7b74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:10.748 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.745+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7403e610 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.748 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.745+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7c1f1250 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.748 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.745+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7c0b4340 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.748 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.746+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b7c0b4fa0 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.952 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.950+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7403e770 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.991 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.989+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7403eb00 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.992 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.989+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f7b7403b050 con 0x7f7b7c0b3e50 2024-03-23T15:47:10.992 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.990+0000 7f7b637fe640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b6806b7b0 0x7f7b6806dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:10.993 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.990+0000 7f7b7affd640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b6806b7b0 0x7f7b6806dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:10.993 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:10.991+0000 7f7b7affd640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b6806b7b0 0x7f7b6806dc70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000f60 tx=0x7f7b70007a80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:11.019 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.016+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f7b7407b9c0 con 0x7f7b7c0b3e50 2024-03-23T15:47:11.019 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.016+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7b74077560 con 0x7f7b7c0b3e50 2024-03-23T15:47:11.198 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.195+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b7c1f1250 con 0x7f7b7c0b3e50 2024-03-23T15:47:11.220 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.217+0000 7f7b637fe640 1 -- 172.21.15.129:0/3462575070 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10945 (secure 0 0 0) 0x7f7b7c1f1250 con 0x7f7b7c0b3e50 2024-03-23T15:47:11.221 INFO:teuthology.misc.health.smithi129.stdout: 2024-03-23T15:47:11.221 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":13,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:10.167917+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b6806b7b0 msgr2=0x7f7b6806dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b6806b7b0 0x7f7b6806dc70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000f60 tx=0x7f7b70007a80 comp rx=0 tx=0).stop 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 msgr2=0x7f7b7c0b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 0x7f7b7c0b30e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009eb0 tx=0x7f7b74004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 shutdown_connections 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b7c0b3e50 0x7f7b7c0b30e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b6806b7b0 0x7f7b6806dc70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b7c15d920 0x7f7b7c151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.223 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 --2- 172.21.15.129:0/3462575070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b7c15d0e0 0x7f7b7c1f0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.224 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 >> 172.21.15.129:0/3462575070 conn(0x7f7b7c0cfb40 msgr2=0x7f7b7c0c7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:11.224 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 shutdown_connections 2024-03-23T15:47:11.224 INFO:teuthology.misc.health.smithi129.stderr:2024-03-23T15:47:11.220+0000 7f7b8138a640 1 -- 172.21.15.129:0/3462575070 wait complete. 2024-03-23T15:47:11.232 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-23T15:47:11.233 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-23T15:47:11.233 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-23T15:47:11.233 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T15:47:11.233 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-03-23T15:47:11.235 INFO:tasks.ceph.mds.a:Started 2024-03-23T15:47:11.235 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-23T15:47:11.236 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T15:47:11.236 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-03-23T15:47:11.238 INFO:tasks.ceph.mds.c:Started 2024-03-23T15:47:11.238 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-23T15:47:11.238 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-23T15:47:11.238 DEBUG:teuthology.orchestra.run.smithi129:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-03-23T15:47:11.240 INFO:tasks.ceph.mds.e:Started 2024-03-23T15:47:11.240 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-23T15:47:11.240 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T15:47:11.240 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-03-23T15:47:11.242 INFO:tasks.ceph.mds.b:Started 2024-03-23T15:47:11.243 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-23T15:47:11.243 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T15:47:11.243 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-03-23T15:47:11.245 INFO:tasks.ceph.mds.d:Started 2024-03-23T15:47:11.245 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-23T15:47:11.245 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-23T15:47:11.245 DEBUG:teuthology.orchestra.run.smithi157:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-03-23T15:47:11.249 INFO:tasks.ceph.mds.f:Started 2024-03-23T15:47:11.250 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-23T15:47:11.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:11.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.353+0000 7fa65c49d640 1 Processor -- start 2024-03-23T15:47:11.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.353+0000 7fa65c49d640 1 -- start start 2024-03-23T15:47:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.354+0000 7fa65c49d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa6540c8290 0x7fa654158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.354+0000 7fa65c49d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65415b5d0 con 0x7fa6541520b0 2024-03-23T15:47:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.354+0000 7fa65c49d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65415c430 con 0x7fa65406f950 2024-03-23T15:47:11.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.354+0000 7fa65c49d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa65406fd60 con 0x7fa6540c8290 2024-03-23T15:47:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.355+0000 7fa65a239640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa6541520b0 0x7fa6540cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38584/0 (socket says 172.21.15.129:38584) 2024-03-23T15:47:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.355+0000 7fa65a239640 1 -- 172.21.15.129:0/2636257076 learned_addr learned my addr 172.21.15.129:0/2636257076 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.355+0000 7fa659a38640 1 --2- 172.21.15.129:0/2636257076 >> v2:172.21.15.157:3300/0 conn(0x7fa6540c8290 0x7fa654158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:11.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.358+0000 7fa659237640 1 -- 172.21.15.129:0/2636257076 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147580072 0 0) 0x7fa65415b5d0 con 0x7fa6541520b0 2024-03-23T15:47:11.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.358+0000 7fa659237640 1 -- 172.21.15.129:0/2636257076 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa640003520 con 0x7fa6541520b0 2024-03-23T15:47:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.360+0000 7fa659a38640 1 -- 172.21.15.129:0/2636257076 >> v1:172.21.15.157:6789/0 conn(0x7fa65406f950 legacy=0x7fa654159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.360+0000 7fa659a38640 1 -- 172.21.15.129:0/2636257076 >> v1:172.21.15.129:6789/0 conn(0x7fa6541520b0 legacy=0x7fa6540cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.360+0000 7fa659a38640 1 -- 172.21.15.129:0/2636257076 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa654152490 con 0x7fa6540c8290 2024-03-23T15:47:11.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.366+0000 7fa659a38640 1 --2- 172.21.15.129:0/2636257076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa654158a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa6440097a0 tx=0x7fa64402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78df9f31df1c5eaf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:11.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa659237640 1 -- 172.21.15.129:0/2636257076 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64402f810 con 0x7fa6540c8290 2024-03-23T15:47:11.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa659237640 1 -- 172.21.15.129:0/2636257076 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa64402f970 con 0x7fa6540c8290 2024-03-23T15:47:11.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa659237640 1 -- 172.21.15.129:0/2636257076 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64402fd00 con 0x7fa6540c8290 2024-03-23T15:47:11.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa65c49d640 1 -- 172.21.15.129:0/2636257076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 msgr2=0x7fa654158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa65c49d640 1 --2- 172.21.15.129:0/2636257076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa654158a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa6440097a0 tx=0x7fa64402b9f0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa65c49d640 1 -- 172.21.15.129:0/2636257076 shutdown_connections 2024-03-23T15:47:11.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa65c49d640 1 --2- 172.21.15.129:0/2636257076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa654158a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.372+0000 7fa65c49d640 1 -- 172.21.15.129:0/2636257076 >> 172.21.15.129:0/2636257076 conn(0x7fa654061b50 msgr2=0x7fa654061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:11.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.373+0000 7fa65c49d640 1 -- 172.21.15.129:0/2636257076 shutdown_connections 2024-03-23T15:47:11.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.373+0000 7fa65c49d640 1 -- 172.21.15.129:0/2636257076 wait complete. 2024-03-23T15:47:11.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.374+0000 7fa65c49d640 1 Processor -- start 2024-03-23T15:47:11.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.374+0000 7fa65c49d640 1 -- start start 2024-03-23T15:47:11.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.374+0000 7fa65c49d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa65406f950 0x7fa6540cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:11.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.374+0000 7fa65c49d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa65406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:11.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.374+0000 7fa65c49d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6541520b0 0x7fa6541f0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:11.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.375+0000 7fa65c49d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa65406d410 con 0x7fa6541520b0 2024-03-23T15:47:11.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.375+0000 7fa65c49d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6541578d0 con 0x7fa6540c8290 2024-03-23T15:47:11.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.375+0000 7fa65c49d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6541eff00 con 0x7fa65406f950 2024-03-23T15:47:11.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.375+0000 7fa65a239640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa65406f950 0x7fa6540cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:11.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.375+0000 7fa659a38640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa65406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:11.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.375+0000 7fa65aa3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6541520b0 0x7fa6541f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:11.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.376+0000 7fa65a239640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa65406f950 0x7fa6540cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:43062/0 (socket says 172.21.15.129:43062) 2024-03-23T15:47:11.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.376+0000 7fa65a239640 1 -- 172.21.15.129:0/71082205 learned_addr learned my addr 172.21.15.129:0/71082205 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:11.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.379+0000 7fa65aa3a640 1 -- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa65406f950 msgr2=0x7fa6540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.379+0000 7fa65aa3a640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa65406f950 0x7fa6540cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.379+0000 7fa65aa3a640 1 -- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 msgr2=0x7fa65406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:11.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.379+0000 7fa65aa3a640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa65406db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:11.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.379+0000 7fa65aa3a640 1 -- 172.21.15.129:0/71082205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa644009400 con 0x7fa6541520b0 2024-03-23T15:47:11.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.381+0000 7fa65aa3a640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6541520b0 0x7fa6541f0390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c6c0 tx=0x7fa64c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:11.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.386+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c003b70 con 0x7fa6541520b0 2024-03-23T15:47:11.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.386+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa64c003cd0 con 0x7fa6541520b0 2024-03-23T15:47:11.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.386+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0052f0 con 0x7fa6541520b0 2024-03-23T15:47:11.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.386+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6541f3340 con 0x7fa6541520b0 2024-03-23T15:47:11.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.386+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6541f38e0 con 0x7fa6541520b0 2024-03-23T15:47:11.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.387+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa654075740 con 0x7fa6541520b0 2024-03-23T15:47:11.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.420+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7fa64c005450 con 0x7fa6541520b0 2024-03-23T15:47:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.420+0000 7fa64affd640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa63806b7d0 0x7fa63806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.420+0000 7fa65a239640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa63806b7d0 0x7fa63806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.421+0000 7fa65a239640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa63806b7d0 0x7fa63806dc90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa6500011e0 tx=0x7fa650009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:11.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.874+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7fa64c059110 con 0x7fa6541520b0 2024-03-23T15:47:11.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:11.874+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa64c0553b0 con 0x7fa6541520b0 2024-03-23T15:47:12.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.157+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6541f0040 con 0x7fa6541520b0 2024-03-23T15:47:12.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.360+0000 7fa64affd640 1 -- 172.21.15.129:0/71082205 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+12924 (secure 0 0 0) 0x7fa64c058a70 con 0x7fa6541520b0 2024-03-23T15:47:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:12.364 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":14,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:11.669584+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa63806b7d0 msgr2=0x7fa63806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa63806b7d0 0x7fa63806dc90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa6500011e0 tx=0x7fa650009040 comp rx=0 tx=0).stop 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6541520b0 msgr2=0x7fa6541f0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6541520b0 0x7fa6541f0390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00c6c0 tx=0x7fa64c00cb80 comp rx=0 tx=0).stop 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 shutdown_connections 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6541520b0 0x7fa6541f0390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa63806b7d0 0x7fa63806dc90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6540c8290 0x7fa65406db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 --2- 172.21.15.129:0/71082205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa65406f950 0x7fa6540cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 >> 172.21.15.129:0/71082205 conn(0x7fa654061b50 msgr2=0x7fa654150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 shutdown_connections 2024-03-23T15:47:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.363+0000 7fa65c49d640 1 -- 172.21.15.129:0/71082205 wait complete. 2024-03-23T15:47:12.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T15:47:11.061293+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T15:47:12.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T15:47:12.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.486+0000 7f00adf63640 1 Processor -- start 2024-03-23T15:47:12.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.486+0000 7f00adf63640 1 -- start start 2024-03-23T15:47:12.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.487+0000 7f00adf63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a81521e0 0x7f00a80c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.487+0000 7f00adf63640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f00a815c110 0x7f00a80c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.487+0000 7f00adf63640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f00a806f8a0 0x7f00a815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.487+0000 7f00adf63640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f00a806fe00 con 0x7f00a815c110 2024-03-23T15:47:12.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.487+0000 7f00adf63640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f00a80c8770 con 0x7f00a806f8a0 2024-03-23T15:47:12.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.487+0000 7f00adf63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00a8070660 con 0x7f00a81521e0 2024-03-23T15:47:12.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.488+0000 7f00a6ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f00a815c110 0x7f00a80c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.488+0000 7f00a7fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f00a806f8a0 0x7f00a815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.488+0000 7f00a77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a81521e0 0x7f00a80c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.488+0000 7f00a6ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f00a815c110 0x7f00a80c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59744/0 (socket says 172.21.15.129:59744) 2024-03-23T15:47:12.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.488+0000 7f00a6ffd640 1 -- 172.21.15.129:0/1266608255 learned_addr learned my addr 172.21.15.129:0/1266608255 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:12.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.490+0000 7f00a6ffd640 1 -- 172.21.15.129:0/1266608255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a81521e0 msgr2=0x7f00a80c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:12.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.490+0000 7f00a6ffd640 1 --2- 172.21.15.129:0/1266608255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a81521e0 0x7f00a80c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.490+0000 7f00a6ffd640 1 -- 172.21.15.129:0/1266608255 >> v2:172.21.15.157:3300/0 conn(0x7f00a806f8a0 msgr2=0x7f00a815b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:12.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.490+0000 7f00a6ffd640 1 --2- 172.21.15.129:0/1266608255 >> v2:172.21.15.157:3300/0 conn(0x7f00a806f8a0 0x7f00a815b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.490+0000 7f00a6ffd640 1 -- 172.21.15.129:0/1266608255 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a8070940 con 0x7f00a815c110 2024-03-23T15:47:12.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.493+0000 7f00a6ffd640 1 --2- 172.21.15.129:0/1266608255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a80c3110 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f00a80704c0 tx=0x7f009c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85216cf496a9755 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:12.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.498+0000 7f00a67fc640 1 -- 172.21.15.129:0/1266608255 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c031870 con 0x7f00a815c110 2024-03-23T15:47:12.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.498+0000 7f00a67fc640 1 -- 172.21.15.129:0/1266608255 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c0319d0 con 0x7f00a815c110 2024-03-23T15:47:12.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.498+0000 7f00a67fc640 1 -- 172.21.15.129:0/1266608255 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c031d60 con 0x7f00a815c110 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 -- 172.21.15.129:0/1266608255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 msgr2=0x7f00a80c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 --2- 172.21.15.129:0/1266608255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a80c3110 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f00a80704c0 tx=0x7f009c02b910 comp rx=0 tx=0).stop 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 -- 172.21.15.129:0/1266608255 shutdown_connections 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 --2- 172.21.15.129:0/1266608255 >> v2:172.21.15.157:3300/0 conn(0x7f00a806f8a0 0x7f00a815b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 --2- 172.21.15.129:0/1266608255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a80c3110 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 --2- 172.21.15.129:0/1266608255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a81521e0 0x7f00a80c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 -- 172.21.15.129:0/1266608255 >> 172.21.15.129:0/1266608255 conn(0x7f00a80718f0 msgr2=0x7f00a8061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 -- 172.21.15.129:0/1266608255 shutdown_connections 2024-03-23T15:47:12.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.499+0000 7f00adf63640 1 -- 172.21.15.129:0/1266608255 wait complete. 2024-03-23T15:47:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.500+0000 7f00adf63640 1 Processor -- start 2024-03-23T15:47:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.500+0000 7f00adf63640 1 -- start start 2024-03-23T15:47:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.501+0000 7f00adf63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a806f8a0 0x7f00a80cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.501+0000 7f00adf63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 0x7f00a8154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.501+0000 7f00adf63640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a814c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.501+0000 7f00adf63640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00a814d700 con 0x7f00a815c110 2024-03-23T15:47:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.501+0000 7f00adf63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f00a80ce770 con 0x7f00a81521e0 2024-03-23T15:47:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.501+0000 7f00adf63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00a81f1140 con 0x7f00a806f8a0 2024-03-23T15:47:12.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.502+0000 7f00a77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a806f8a0 0x7f00a80cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.502+0000 7f00a6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 0x7f00a8154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.502+0000 7f00a6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 0x7f00a8154f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:49016/0 (socket says 172.21.15.129:49016) 2024-03-23T15:47:12.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.502+0000 7f00a6ffd640 1 -- 172.21.15.129:0/2368573884 learned_addr learned my addr 172.21.15.129:0/2368573884 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:12.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.502+0000 7f00a7fff640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a814c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.506+0000 7f00a6ffd640 1 -- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a806f8a0 msgr2=0x7f00a80cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.506+0000 7f00a6ffd640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a806f8a0 0x7f00a80cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.506+0000 7f00a6ffd640 1 -- 172.21.15.129:0/2368573884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 msgr2=0x7f00a814c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.506+0000 7f00a6ffd640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a814c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.506+0000 7f00a6ffd640 1 -- 172.21.15.129:0/2368573884 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c0094d0 con 0x7f00a81521e0 2024-03-23T15:47:12.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.509+0000 7f00a6ffd640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 0x7f00a8154f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f009c000c00 tx=0x7f009c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:12.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.513+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c03ac60 con 0x7f00a81521e0 2024-03-23T15:47:12.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.513+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c03adc0 con 0x7f00a81521e0 2024-03-23T15:47:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.513+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c044630 con 0x7f00a81521e0 2024-03-23T15:47:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.513+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a80ce490 con 0x7f00a81521e0 2024-03-23T15:47:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.514+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a814c120 con 0x7f00a81521e0 2024-03-23T15:47:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.515+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a8075740 con 0x7f00a81521e0 2024-03-23T15:47:12.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.538+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f009c0303c0 con 0x7f00a81521e0 2024-03-23T15:47:12.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.539+0000 7f008ffff640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f009406b800 0x7f009406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:12.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.539+0000 7f00a77fe640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f009406b800 0x7f009406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:12.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.540+0000 7f00a77fe640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f009406b800 0x7f009406dcc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0098009b40 tx=0x7f0098009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.779+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4846+0+0 (secure 0 0 0) 0x7f009c02d040 con 0x7f00a81521e0 2024-03-23T15:47:12.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.801+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f009c077870 con 0x7f00a81521e0 2024-03-23T15:47:12.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:12.987+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00a80c68b0 con 0x7f00a81521e0 2024-03-23T15:47:13.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.055+0000 7f008ffff640 1 -- 172.21.15.129:0/2368573884 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v14) ==== 93+0+10 (secure 0 0 0) 0x7f009c07af30 con 0x7f00a81521e0 2024-03-23T15:47:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f009406b800 msgr2=0x7f009406dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f009406b800 0x7f009406dcc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0098009b40 tx=0x7f0098009190 comp rx=0 tx=0).stop 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 msgr2=0x7f00a8154f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 0x7f00a8154f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f009c000c00 tx=0x7f009c0046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 shutdown_connections 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00a815c110 0x7f00a814c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f009406b800 0x7f009406dcc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81521e0 0x7f00a8154f80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 --2- 172.21.15.129:0/2368573884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a806f8a0 0x7f00a80cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 >> 172.21.15.129:0/2368573884 conn(0x7f00a80718f0 msgr2=0x7f00a8159eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 shutdown_connections 2024-03-23T15:47:13.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.058+0000 7f00adf63640 1 -- 172.21.15.129:0/2368573884 wait complete. 2024-03-23T15:47:13.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:13.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.179+0000 7f78e3363640 1 Processor -- start 2024-03-23T15:47:13.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.179+0000 7f78e3363640 1 -- start start 2024-03-23T15:47:13.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.180+0000 7f78e3363640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f78dc1520b0 0x7f78dc158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:13.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.180+0000 7f78e3363640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:13.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.180+0000 7f78e3363640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78dc15b5d0 con 0x7f78dc0c8290 2024-03-23T15:47:13.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.180+0000 7f78e3363640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f78dc15c430 con 0x7f78dc1520b0 2024-03-23T15:47:13.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.180+0000 7f78e3363640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78dc06fd60 con 0x7f78dc06f950 2024-03-23T15:47:13.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.190+0000 7f78e1900640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:13.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.190+0000 7f78e1900640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:43094/0 (socket says 172.21.15.129:43094) 2024-03-23T15:47:13.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.190+0000 7f78e1900640 1 -- 172.21.15.129:0/3929972226 learned_addr learned my addr 172.21.15.129:0/3929972226 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:13.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.193+0000 7f78e1900640 1 -- 172.21.15.129:0/3929972226 >> v2:172.21.15.129:3300/0 conn(0x7f78dc1520b0 msgr2=0x7f78dc158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:13.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.193+0000 7f78e1900640 1 --2- 172.21.15.129:0/3929972226 >> v2:172.21.15.129:3300/0 conn(0x7f78dc1520b0 0x7f78dc158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.193+0000 7f78e1900640 1 -- 172.21.15.129:0/3929972226 >> v1:172.21.15.157:6789/0 conn(0x7f78dc0c8290 legacy=0x7f78dc0cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.193+0000 7f78e1900640 1 -- 172.21.15.129:0/3929972226 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78dc152490 con 0x7f78dc06f950 2024-03-23T15:47:13.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.195+0000 7f78e1900640 1 --2- 172.21.15.129:0/3929972226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc159680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009e50 tx=0x7f78d402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba816b0a2339099 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:13.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.199+0000 7f78d3fff640 1 -- 172.21.15.129:0/3929972226 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d402cb00 con 0x7f78dc06f950 2024-03-23T15:47:13.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.199+0000 7f78d3fff640 1 -- 172.21.15.129:0/3929972226 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78d402cc60 con 0x7f78dc06f950 2024-03-23T15:47:13.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.199+0000 7f78d3fff640 1 -- 172.21.15.129:0/3929972226 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d40343f0 con 0x7f78dc06f950 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 -- 172.21.15.129:0/3929972226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 msgr2=0x7f78dc159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 --2- 172.21.15.129:0/3929972226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc159680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009e50 tx=0x7f78d402bf60 comp rx=0 tx=0).stop 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 -- 172.21.15.129:0/3929972226 shutdown_connections 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 --2- 172.21.15.129:0/3929972226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc159680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 --2- 172.21.15.129:0/3929972226 >> v2:172.21.15.129:3300/0 conn(0x7f78dc1520b0 0x7f78dc158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 -- 172.21.15.129:0/3929972226 >> 172.21.15.129:0/3929972226 conn(0x7f78dc061b50 msgr2=0x7f78dc061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 -- 172.21.15.129:0/3929972226 shutdown_connections 2024-03-23T15:47:13.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.200+0000 7f78e3363640 1 -- 172.21.15.129:0/3929972226 wait complete. 2024-03-23T15:47:13.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.202+0000 7f78e3363640 1 Processor -- start 2024-03-23T15:47:13.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.202+0000 7f78e3363640 1 -- start start 2024-03-23T15:47:13.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.203+0000 7f78e3363640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc0c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:13.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.203+0000 7f78e3363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78dc0c8290 0x7f78dc0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:13.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.203+0000 7f78e3363640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78dc1520b0 0x7f78dc14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:13.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.203+0000 7f78e3363640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc0c5440 con 0x7f78dc1520b0 2024-03-23T15:47:13.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.203+0000 7f78e3363640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc0c7480 con 0x7f78dc0c8290 2024-03-23T15:47:13.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.203+0000 7f78e3363640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78dc06d8a0 con 0x7f78dc06f950 2024-03-23T15:47:13.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.204+0000 7f78e10ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc0c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:13.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.204+0000 7f78e10ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc0c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:43096/0 (socket says 172.21.15.129:43096) 2024-03-23T15:47:13.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.204+0000 7f78e10ff640 1 -- 172.21.15.129:0/2146236743 learned_addr learned my addr 172.21.15.129:0/2146236743 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:13.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.207+0000 7f78e10ff640 1 -- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78dc0c8290 msgr2=0x7f78dc0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.207+0000 7f78e10ff640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78dc0c8290 0x7f78dc0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.207+0000 7f78e10ff640 1 -- 172.21.15.129:0/2146236743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78dc1520b0 msgr2=0x7f78dc14e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.207+0000 7f78e10ff640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78dc1520b0 0x7f78dc14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.207+0000 7f78e10ff640 1 -- 172.21.15.129:0/2146236743 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d4009b30 con 0x7f78dc06f950 2024-03-23T15:47:13.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.210+0000 7f78e10ff640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc0c4e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f78d800b4a0 tx=0x7f78d800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.212+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8010610 con 0x7f78dc06f950 2024-03-23T15:47:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.212+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78dc06db80 con 0x7f78dc06f950 2024-03-23T15:47:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.212+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78dc06e140 con 0x7f78dc06f950 2024-03-23T15:47:13.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.213+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78dc0686f0 con 0x7f78dc06f950 2024-03-23T15:47:13.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.214+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78d8010770 con 0x7f78dc06f950 2024-03-23T15:47:13.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.214+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8010b40 con 0x7f78dc06f950 2024-03-23T15:47:13.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.249+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f78d8010ca0 con 0x7f78dc06f950 2024-03-23T15:47:13.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.250+0000 7f78d1ffb640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78c006b780 0x7f78c006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:13.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.250+0000 7f78e08fe640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78c006b780 0x7f78c006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:13.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.251+0000 7f78e08fe640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78c006b780 0x7f78c006dc40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f78dc062240 tx=0x7f78cc007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.500+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 4846+0+0 (secure 0 0 0) 0x7f78d80590f0 con 0x7f78dc06f950 2024-03-23T15:47:13.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.511+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f78d8054c90 con 0x7f78dc06f950 2024-03-23T15:47:13.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.747+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78dc075740 con 0x7f78dc06f950 2024-03-23T15:47:13.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.774+0000 7f78d1ffb640 1 -- 172.21.15.129:0/2146236743 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f78d801e070 con 0x7f78dc06f950 2024-03-23T15:47:13.776 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:13.776 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-23T15:47:13.776 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 1 2024-03-23T15:47:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78c006b780 msgr2=0x7f78c006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78c006b780 0x7f78c006dc40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f78dc062240 tx=0x7f78cc007990 comp rx=0 tx=0).stop 2024-03-23T15:47:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 msgr2=0x7f78dc0c4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc0c4e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f78d800b4a0 tx=0x7f78d800b960 comp rx=0 tx=0).stop 2024-03-23T15:47:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 shutdown_connections 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78dc1520b0 0x7f78dc14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78c006b780 0x7f78c006dc40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78dc0c8290 0x7f78dc0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 --2- 172.21.15.129:0/2146236743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78dc06f950 0x7f78dc0c4e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 >> 172.21.15.129:0/2146236743 conn(0x7f78dc061b50 msgr2=0x7f78dc0cc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 shutdown_connections 2024-03-23T15:47:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:13.776+0000 7f78e3363640 1 -- 172.21.15.129:0/2146236743 wait complete. 2024-03-23T15:47:15.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:15.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.901+0000 7fd647f6c640 1 Processor -- start 2024-03-23T15:47:15.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.901+0000 7fd647f6c640 1 -- start start 2024-03-23T15:47:15.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.902+0000 7fd647f6c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd6400c8290 0x7fd640158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:15.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.902+0000 7fd647f6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd640159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:15.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.902+0000 7fd647f6c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd64015b5d0 con 0x7fd6401520b0 2024-03-23T15:47:15.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.902+0000 7fd647f6c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd64015c430 con 0x7fd6400c8290 2024-03-23T15:47:15.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.902+0000 7fd647f6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd64006fd60 con 0x7fd64006f950 2024-03-23T15:47:15.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.903+0000 7fd645d08640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd6401520b0 0x7fd6400cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38596/0 (socket says 172.21.15.129:38596) 2024-03-23T15:47:15.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.903+0000 7fd645d08640 1 -- 172.21.15.129:0/3109075875 learned_addr learned my addr 172.21.15.129:0/3109075875 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:15.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.903+0000 7fd646509640 1 --2- 172.21.15.129:0/3109075875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd640159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:15.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.905+0000 7fd645507640 1 --2- 172.21.15.129:0/3109075875 >> v2:172.21.15.157:3300/0 conn(0x7fd6400c8290 0x7fd640158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:15.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.906+0000 7fd644d06640 1 -- 172.21.15.129:0/3109075875 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594308649 0 0) 0x7fd64015b5d0 con 0x7fd6401520b0 2024-03-23T15:47:15.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.906+0000 7fd644d06640 1 -- 172.21.15.129:0/3109075875 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd62c003520 con 0x7fd6401520b0 2024-03-23T15:47:15.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.907+0000 7fd646509640 1 -- 172.21.15.129:0/3109075875 >> v2:172.21.15.157:3300/0 conn(0x7fd6400c8290 msgr2=0x7fd640158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:15.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.907+0000 7fd646509640 1 --2- 172.21.15.129:0/3109075875 >> v2:172.21.15.157:3300/0 conn(0x7fd6400c8290 0x7fd640158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:15.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.907+0000 7fd646509640 1 -- 172.21.15.129:0/3109075875 >> v1:172.21.15.129:6789/0 conn(0x7fd6401520b0 legacy=0x7fd6400cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:15.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.907+0000 7fd646509640 1 -- 172.21.15.129:0/3109075875 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd640152490 con 0x7fd64006f950 2024-03-23T15:47:15.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.911+0000 7fd646509640 1 --2- 172.21.15.129:0/3109075875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd640159680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd638009d90 tx=0x7fd63802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=454c8d27b45d5199 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.916+0000 7fd644d06640 1 -- 172.21.15.129:0/3109075875 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63802cce0 con 0x7fd64006f950 2024-03-23T15:47:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.916+0000 7fd644d06640 1 -- 172.21.15.129:0/3109075875 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd63802ce40 con 0x7fd64006f950 2024-03-23T15:47:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.916+0000 7fd644d06640 1 -- 172.21.15.129:0/3109075875 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638034660 con 0x7fd64006f950 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 -- 172.21.15.129:0/3109075875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 msgr2=0x7fd640159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 --2- 172.21.15.129:0/3109075875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd640159680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd638009d90 tx=0x7fd63802bf30 comp rx=0 tx=0).stop 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 -- 172.21.15.129:0/3109075875 shutdown_connections 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 --2- 172.21.15.129:0/3109075875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd640159680 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 --2- 172.21.15.129:0/3109075875 >> v2:172.21.15.157:3300/0 conn(0x7fd6400c8290 0x7fd640158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 -- 172.21.15.129:0/3109075875 >> 172.21.15.129:0/3109075875 conn(0x7fd640061b50 msgr2=0x7fd640061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 -- 172.21.15.129:0/3109075875 shutdown_connections 2024-03-23T15:47:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.917+0000 7fd647f6c640 1 -- 172.21.15.129:0/3109075875 wait complete. 2024-03-23T15:47:15.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.919+0000 7fd647f6c640 1 Processor -- start 2024-03-23T15:47:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.919+0000 7fd647f6c640 1 -- start start 2024-03-23T15:47:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.920+0000 7fd647f6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd6401ef970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.920+0000 7fd647f6c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6400c8290 0x7fd64006d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.920+0000 7fd647f6c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 0x7fd640153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.920+0000 7fd647f6c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd64006c5f0 con 0x7fd6401520b0 2024-03-23T15:47:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.920+0000 7fd647f6c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd64006c8d0 con 0x7fd6400c8290 2024-03-23T15:47:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.920+0000 7fd647f6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd64006cbb0 con 0x7fd64006f950 2024-03-23T15:47:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.921+0000 7fd646509640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 0x7fd640153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.921+0000 7fd645d08640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd6401ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.921+0000 7fd645507640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6400c8290 0x7fd64006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.921+0000 7fd646509640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 0x7fd640153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59776/0 (socket says 172.21.15.129:59776) 2024-03-23T15:47:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.921+0000 7fd646509640 1 -- 172.21.15.129:0/2080788394 learned_addr learned my addr 172.21.15.129:0/2080788394 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:15.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.926+0000 7fd646509640 1 -- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 msgr2=0x7fd6401ef970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:15.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.926+0000 7fd646509640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd6401ef970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:15.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.926+0000 7fd646509640 1 -- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6400c8290 msgr2=0x7fd64006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:15.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.926+0000 7fd646509640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6400c8290 0x7fd64006d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:15.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.926+0000 7fd646509640 1 -- 172.21.15.129:0/2080788394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd638009a70 con 0x7fd6401520b0 2024-03-23T15:47:15.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.929+0000 7fd646509640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 0x7fd640153ba0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd638009d90 tx=0x7fd638004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:15.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.933+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63803d600 con 0x7fd6401520b0 2024-03-23T15:47:15.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.933+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd63803d780 con 0x7fd6401520b0 2024-03-23T15:47:15.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.933+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64006ce90 con 0x7fd6401520b0 2024-03-23T15:47:15.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.933+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63803db30 con 0x7fd6401520b0 2024-03-23T15:47:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.934+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64006d450 con 0x7fd6401520b0 2024-03-23T15:47:15.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.935+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd640075740 con 0x7fd6401520b0 2024-03-23T15:47:15.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.964+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7fd63803b040 con 0x7fd6401520b0 2024-03-23T15:47:15.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.964+0000 7fd6367fc640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62406b780 0x7fd62406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:15.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.964+0000 7fd645d08640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62406b780 0x7fd62406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:15.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:15.965+0000 7fd645d08640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62406b780 0x7fd62406dc40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd63c005880 tx=0x7fd63c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.219+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd63807b210 con 0x7fd6401520b0 2024-03-23T15:47:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.219+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd6380774b0 con 0x7fd6401520b0 2024-03-23T15:47:16.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.506+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6400c7c10 con 0x7fd6401520b0 2024-03-23T15:47:16.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.734+0000 7fd6367fc640 1 -- 172.21.15.129:0/2080788394 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fd63804e090 con 0x7fd6401520b0 2024-03-23T15:47:16.737 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:16.737 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-23T15:47:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 1 2024-03-23T15:47:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62406b780 msgr2=0x7fd62406dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62406b780 0x7fd62406dc40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd63c005880 tx=0x7fd63c009040 comp rx=0 tx=0).stop 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 msgr2=0x7fd640153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 0x7fd640153ba0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd638009d90 tx=0x7fd638004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 shutdown_connections 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6401520b0 0x7fd640153ba0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62406b780 0x7fd62406dc40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6400c8290 0x7fd64006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 --2- 172.21.15.129:0/2080788394 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006f950 0x7fd6401ef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 >> 172.21.15.129:0/2080788394 conn(0x7fd640061b50 msgr2=0x7fd640150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 shutdown_connections 2024-03-23T15:47:16.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:16.736+0000 7fd647f6c640 1 -- 172.21.15.129:0/2080788394 wait complete. 2024-03-23T15:47:17.767 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-03-23T15:47:17.830 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-03-23T15:47:17.838 INFO:tasks.ceph.mds.f.smithi157.stdout:starting mds.f at 2024-03-23T15:47:17.993 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-03-23T15:47:18.086 INFO:tasks.ceph.mds.e.smithi129.stdout:starting mds.e at 2024-03-23T15:47:18.119 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-03-23T15:47:18.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:18.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.849+0000 7fdd87df4640 1 Processor -- start 2024-03-23T15:47:18.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.849+0000 7fdd87df4640 1 -- start start 2024-03-23T15:47:18.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.850+0000 7fdd87df4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdd801545a0 0x7fdd80071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:18.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.850+0000 7fdd87df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 0x7fdd8015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:18.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.850+0000 7fdd87df4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd80072ce0 con 0x7fdd8015e2f0 2024-03-23T15:47:18.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.850+0000 7fdd87df4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdd800c15c0 con 0x7fdd801545a0 2024-03-23T15:47:18.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.850+0000 7fdd87df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8015e780 con 0x7fdd80072030 2024-03-23T15:47:18.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.851+0000 7fdd85b90640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdd801545a0 0x7fdd80071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:18.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.851+0000 7fdd8538f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 0x7fdd8015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:18.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.851+0000 7fdd8538f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 0x7fdd8015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:43186/0 (socket says 172.21.15.129:43186) 2024-03-23T15:47:18.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.851+0000 7fdd8538f640 1 -- 172.21.15.129:0/1340996263 learned_addr learned my addr 172.21.15.129:0/1340996263 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.854+0000 7fdd8538f640 1 -- 172.21.15.129:0/1340996263 >> v2:172.21.15.157:3300/0 conn(0x7fdd801545a0 msgr2=0x7fdd80071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.854+0000 7fdd8538f640 1 --2- 172.21.15.129:0/1340996263 >> v2:172.21.15.157:3300/0 conn(0x7fdd801545a0 0x7fdd80071af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.854+0000 7fdd8538f640 1 -- 172.21.15.129:0/1340996263 >> v1:172.21.15.157:6789/0 conn(0x7fdd8015e2f0 legacy=0x7fdd8006b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:18.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.854+0000 7fdd8538f640 1 -- 172.21.15.129:0/1340996263 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8015a5f0 con 0x7fdd80072030 2024-03-23T15:47:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.856+0000 7fdd8538f640 1 --2- 172.21.15.129:0/1340996263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 0x7fdd8015ddb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009870 tx=0x7fdd7002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e97dac5ecdec5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd84b8e640 1 -- 172.21.15.129:0/1340996263 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70031640 con 0x7fdd80072030 2024-03-23T15:47:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd84b8e640 1 -- 172.21.15.129:0/1340996263 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd700317a0 con 0x7fdd80072030 2024-03-23T15:47:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd84b8e640 1 -- 172.21.15.129:0/1340996263 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70031b30 con 0x7fdd80072030 2024-03-23T15:47:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 -- 172.21.15.129:0/1340996263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 msgr2=0x7fdd8015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 --2- 172.21.15.129:0/1340996263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 0x7fdd8015ddb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009870 tx=0x7fdd7002b8f0 comp rx=0 tx=0).stop 2024-03-23T15:47:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 -- 172.21.15.129:0/1340996263 shutdown_connections 2024-03-23T15:47:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 --2- 172.21.15.129:0/1340996263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd80072030 0x7fdd8015ddb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 --2- 172.21.15.129:0/1340996263 >> v2:172.21.15.157:3300/0 conn(0x7fdd801545a0 0x7fdd80071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 -- 172.21.15.129:0/1340996263 >> 172.21.15.129:0/1340996263 conn(0x7fdd800cbb20 msgr2=0x7fdd800cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 -- 172.21.15.129:0/1340996263 shutdown_connections 2024-03-23T15:47:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.860+0000 7fdd87df4640 1 -- 172.21.15.129:0/1340996263 wait complete. 2024-03-23T15:47:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.861+0000 7fdd87df4640 1 Processor -- start 2024-03-23T15:47:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.861+0000 7fdd87df4640 1 -- start start 2024-03-23T15:47:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.862+0000 7fdd87df4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd80072030 0x7fdd801476d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.862+0000 7fdd87df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 0x7fdd80148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.862+0000 7fdd87df4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd8015e2f0 0x7fdd8015c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.862+0000 7fdd87df4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801f0380 con 0x7fdd80072030 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.862+0000 7fdd87df4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801496e0 con 0x7fdd8015e2f0 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.862+0000 7fdd87df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd801f0be0 con 0x7fdd801545a0 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.863+0000 7fdd8538f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 0x7fdd80148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.863+0000 7fdd86391640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd8015e2f0 0x7fdd8015c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.863+0000 7fdd85b90640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd80072030 0x7fdd801476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.863+0000 7fdd8538f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 0x7fdd80148380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:43200/0 (socket says 172.21.15.129:43200) 2024-03-23T15:47:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.863+0000 7fdd8538f640 1 -- 172.21.15.129:0/117664863 learned_addr learned my addr 172.21.15.129:0/117664863 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:18.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.866+0000 7fdd8538f640 1 -- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd8015e2f0 msgr2=0x7fdd8015c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:18.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.866+0000 7fdd8538f640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd8015e2f0 0x7fdd8015c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:18.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.866+0000 7fdd8538f640 1 -- 172.21.15.129:0/117664863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd80072030 msgr2=0x7fdd801476d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:18.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.866+0000 7fdd8538f640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd80072030 0x7fdd801476d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:18.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.866+0000 7fdd8538f640 1 -- 172.21.15.129:0/117664863 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd700094d0 con 0x7fdd801545a0 2024-03-23T15:47:18.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.869+0000 7fdd8538f640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 0x7fdd80148380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd70000c00 tx=0x7fdd700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:18.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.871+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7003a930 con 0x7fdd801545a0 2024-03-23T15:47:18.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.871+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd800c9ab0 con 0x7fdd801545a0 2024-03-23T15:47:18.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.871+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd80149400 con 0x7fdd801545a0 2024-03-23T15:47:18.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.872+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd80075740 con 0x7fdd801545a0 2024-03-23T15:47:18.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.873+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd7003aa90 con 0x7fdd801545a0 2024-03-23T15:47:18.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:18.873+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7003ae00 con 0x7fdd801545a0 2024-03-23T15:47:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.122+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd7002f050 con 0x7fdd801545a0 2024-03-23T15:47:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.123+0000 7fdd767fc640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd6406b810 0x7fdd6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.123+0000 7fdd85b90640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd6406b810 0x7fdd6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.123+0000 7fdd85b90640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd6406b810 0x7fdd6406dcd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0113e0 tx=0x7fdd7c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:19.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.126+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fdd7007ace0 con 0x7fdd801545a0 2024-03-23T15:47:19.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.137+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd7007aab0 con 0x7fdd801545a0 2024-03-23T15:47:19.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.345+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd80068ac0 con 0x7fdd801545a0 2024-03-23T15:47:19.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.454+0000 7fdd767fc640 1 -- 172.21.15.129:0/117664863 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7fdd80068ac0 con 0x7fdd801545a0 2024-03-23T15:47:19.457 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:19.458 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-23T15:47:19.458 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 2 2024-03-23T15:47:19.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd6406b810 msgr2=0x7fdd6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd6406b810 0x7fdd6406dcd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0113e0 tx=0x7fdd7c00b040 comp rx=0 tx=0).stop 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 msgr2=0x7fdd80148380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 0x7fdd80148380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd70000c00 tx=0x7fdd700046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 shutdown_connections 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd6406b810 0x7fdd6406dcd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd8015e2f0 0x7fdd8015c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd801545a0 0x7fdd80148380 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.456+0000 7fdd87df4640 1 --2- 172.21.15.129:0/117664863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd80072030 0x7fdd801476d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.457+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 >> 172.21.15.129:0/117664863 conn(0x7fdd800cbb20 msgr2=0x7fdd800c85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.457+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 shutdown_connections 2024-03-23T15:47:19.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:19.457+0000 7fdd87df4640 1 -- 172.21.15.129:0/117664863 wait complete. 2024-03-23T15:47:21.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:21.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.571+0000 7fcd20fb5640 1 Processor -- start 2024-03-23T15:47:21.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.571+0000 7fcd20fb5640 1 -- start start 2024-03-23T15:47:21.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.572+0000 7fcd20fb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:21.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.572+0000 7fcd20fb5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcd1c156f60 0x7fcd1c157370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.572+0000 7fcd20fb5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd1c0c3bd0 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.572+0000 7fcd20fb5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1c0c7ca0 con 0x7fcd1c156f60 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.572+0000 7fcd20fb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1c0c57d0 con 0x7fcd1c0c7380 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.573+0000 7fcd1ad9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcd1c156f60 0x7fcd1c157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.573+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.573+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59264/0 (socket says 172.21.15.129:59264) 2024-03-23T15:47:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.573+0000 7fcd1a59c640 1 -- 172.21.15.129:0/349912437 learned_addr learned my addr 172.21.15.129:0/349912437 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.575+0000 7fcd1a59c640 1 -- 172.21.15.129:0/349912437 >> v2:172.21.15.129:3300/0 conn(0x7fcd1c156f60 msgr2=0x7fcd1c157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.575+0000 7fcd1a59c640 1 --2- 172.21.15.129:0/349912437 >> v2:172.21.15.129:3300/0 conn(0x7fcd1c156f60 0x7fcd1c157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.575+0000 7fcd1a59c640 1 -- 172.21.15.129:0/349912437 >> v1:172.21.15.129:6789/0 conn(0x7fcd1c0c8e90 legacy=0x7fcd1c0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.575+0000 7fcd1a59c640 1 -- 172.21.15.129:0/349912437 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c0c3740 con 0x7fcd1c0c7380 2024-03-23T15:47:21.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.577+0000 7fcd1a59c640 1 --2- 172.21.15.129:0/349912437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0c7760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400bb10 tx=0x7fcd1402e1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5eb9b26e6833a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.581+0000 7fcd1959a640 1 -- 172.21.15.129:0/349912437 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14036440 con 0x7fcd1c0c7380 2024-03-23T15:47:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.581+0000 7fcd1959a640 1 -- 172.21.15.129:0/349912437 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd140365a0 con 0x7fcd1c0c7380 2024-03-23T15:47:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.581+0000 7fcd1959a640 1 -- 172.21.15.129:0/349912437 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14036930 con 0x7fcd1c0c7380 2024-03-23T15:47:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.581+0000 7fcd20fb5640 1 -- 172.21.15.129:0/349912437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 msgr2=0x7fcd1c0c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.581+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/349912437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0c7760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400bb10 tx=0x7fcd1402e1f0 comp rx=0 tx=0).stop 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.582+0000 7fcd20fb5640 1 -- 172.21.15.129:0/349912437 shutdown_connections 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.582+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/349912437 >> v2:172.21.15.129:3300/0 conn(0x7fcd1c156f60 0x7fcd1c157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.582+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/349912437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0c7760 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.582+0000 7fcd20fb5640 1 -- 172.21.15.129:0/349912437 >> 172.21.15.129:0/349912437 conn(0x7fcd1c0c8370 msgr2=0x7fcd1c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.582+0000 7fcd20fb5640 1 -- 172.21.15.129:0/349912437 shutdown_connections 2024-03-23T15:47:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.582+0000 7fcd20fb5640 1 -- 172.21.15.129:0/349912437 wait complete. 2024-03-23T15:47:21.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.583+0000 7fcd20fb5640 1 Processor -- start 2024-03-23T15:47:21.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.583+0000 7fcd20fb5640 1 -- start start 2024-03-23T15:47:21.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.584+0000 7fcd20fb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:21.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.584+0000 7fcd20fb5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd1c0c8e90 0x7fcd1c144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:21.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.584+0000 7fcd20fb5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd1c156f60 0x7fcd1c13d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:21.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.584+0000 7fcd20fb5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c0cb6f0 con 0x7fcd1c156f60 2024-03-23T15:47:21.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.584+0000 7fcd20fb5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c140220 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.584+0000 7fcd20fb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1c1558a0 con 0x7fcd1c0c7380 2024-03-23T15:47:21.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.585+0000 7fcd1ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd1c156f60 0x7fcd1c13d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:21.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.585+0000 7fcd1ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd1c156f60 0x7fcd1c13d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37374/0 (socket says 172.21.15.129:37374) 2024-03-23T15:47:21.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.585+0000 7fcd1ad9d640 1 -- 172.21.15.129:0/1978707878 learned_addr learned my addr 172.21.15.129:0/1978707878 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:21.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.585+0000 7fcd19d9b640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd1c0c8e90 0x7fcd1c144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:21.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.586+0000 7fcd1a59c640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:21.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.589+0000 7fcd19d9b640 1 -- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 msgr2=0x7fcd1c0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:21.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.589+0000 7fcd19d9b640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:21.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.589+0000 7fcd19d9b640 1 -- 172.21.15.129:0/1978707878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd1c156f60 msgr2=0x7fcd1c13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:21.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.589+0000 7fcd19d9b640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd1c156f60 0x7fcd1c13d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:21.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.589+0000 7fcd19d9b640 1 -- 172.21.15.129:0/1978707878 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1400b770 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.591+0000 7fcd19d9b640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd1c0c8e90 0x7fcd1c144ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c340 tx=0x7fcd0400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.595+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd040052d0 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.595+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd04005450 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.595+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd1c0ca6c0 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.595+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04007260 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.595+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1c154f70 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.596+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd1c155a50 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.615+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcd040073c0 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.615+0000 7fcd12ffd640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fccfc06b890 0x7fccfc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:21.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.616+0000 7fcd1a59c640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fccfc06b890 0x7fccfc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:21.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.616+0000 7fcd1a59c640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fccfc06b890 0x7fccfc06dd50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd14038000 tx=0x7fcd14032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:21.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.847+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fcd04058330 con 0x7fcd1c0c8e90 2024-03-23T15:47:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:21.857+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcd040545d0 con 0x7fcd1c0c8e90 2024-03-23T15:47:22.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.066+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd1c155a10 con 0x7fcd1c0c8e90 2024-03-23T15:47:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.084+0000 7fcd12ffd640 1 -- 172.21.15.129:0/1978707878 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5527 (secure 0 0 0) 0x7fcd04020020 con 0x7fcd1c0c8e90 2024-03-23T15:47:22.087 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:22.087 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4234,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4246,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4250,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3877606926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3877606926},{"type":"v1","addr":"172.21.15.157:6837","nonce":3877606926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4253,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-23T15:47:22.088 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 3 2024-03-23T15:47:22.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fccfc06b890 msgr2=0x7fccfc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fccfc06b890 0x7fccfc06dd50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd14038000 tx=0x7fcd14032000 comp rx=0 tx=0).stop 2024-03-23T15:47:22.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd1c0c8e90 msgr2=0x7fcd1c144ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd1c0c8e90 0x7fcd1c144ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c340 tx=0x7fcd0400c800 comp rx=0 tx=0).stop 2024-03-23T15:47:22.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 shutdown_connections 2024-03-23T15:47:22.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd1c156f60 0x7fcd1c13d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fccfc06b890 0x7fccfc06dd50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd1c0c8e90 0x7fcd1c144ad0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 --2- 172.21.15.129:0/1978707878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd1c0c7380 0x7fcd1c0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 >> 172.21.15.129:0/1978707878 conn(0x7fcd1c0c8370 msgr2=0x7fcd1c06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:22.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 shutdown_connections 2024-03-23T15:47:22.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.086+0000 7fcd20fb5640 1 -- 172.21.15.129:0/1978707878 wait complete. 2024-03-23T15:47:22.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:22.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.199+0000 7facfe86b640 1 Processor -- start 2024-03-23T15:47:22.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.199+0000 7facfe86b640 1 -- start start 2024-03-23T15:47:22.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.200+0000 7facfe86b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facf815dc20 0x7facf815e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.200+0000 7facfe86b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7facf80c6b20 0x7facf80ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.200+0000 7facfe86b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf80ad9b0 con 0x7facf815e540 2024-03-23T15:47:22.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.200+0000 7facfe86b640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7facf80c74e0 con 0x7facf80c6b20 2024-03-23T15:47:22.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.200+0000 7facfe86b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7facf80c8a90 con 0x7facf815dc20 2024-03-23T15:47:22.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.201+0000 7facf7fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facf815dc20 0x7facf815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:22.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.201+0000 7facf7fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facf815dc20 0x7facf815e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52616/0 (socket says 172.21.15.129:52616) 2024-03-23T15:47:22.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.201+0000 7facf7fff640 1 -- 172.21.15.129:0/2718842159 learned_addr learned my addr 172.21.15.129:0/2718842159 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.203+0000 7facf7fff640 1 -- 172.21.15.129:0/2718842159 >> v2:172.21.15.129:3300/0 conn(0x7facf80c6b20 msgr2=0x7facf80ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.203+0000 7facf7fff640 1 --2- 172.21.15.129:0/2718842159 >> v2:172.21.15.129:3300/0 conn(0x7facf80c6b20 0x7facf80ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.203+0000 7facf7fff640 1 -- 172.21.15.129:0/2718842159 >> v1:172.21.15.129:6789/0 conn(0x7facf815e540 legacy=0x7facf81541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.203+0000 7facf7fff640 1 -- 172.21.15.129:0/2718842159 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf80c87b0 con 0x7facf815dc20 2024-03-23T15:47:22.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.205+0000 7facf7fff640 1 --2- 172.21.15.129:0/2718842159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf815dc20 0x7facf815e000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7facf806cf30 tx=0x7facec02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1b51762e952dd08 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:22.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facf6ffd640 1 -- 172.21.15.129:0/2718842159 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec030860 con 0x7facf815dc20 2024-03-23T15:47:22.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facf6ffd640 1 -- 172.21.15.129:0/2718842159 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec0309c0 con 0x7facf815dc20 2024-03-23T15:47:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facf6ffd640 1 -- 172.21.15.129:0/2718842159 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec030d50 con 0x7facf815dc20 2024-03-23T15:47:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 -- 172.21.15.129:0/2718842159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf815dc20 msgr2=0x7facf815e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 --2- 172.21.15.129:0/2718842159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf815dc20 0x7facf815e000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7facf806cf30 tx=0x7facec02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 -- 172.21.15.129:0/2718842159 shutdown_connections 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 --2- 172.21.15.129:0/2718842159 >> v2:172.21.15.129:3300/0 conn(0x7facf80c6b20 0x7facf80ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 --2- 172.21.15.129:0/2718842159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf815dc20 0x7facf815e000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 -- 172.21.15.129:0/2718842159 >> 172.21.15.129:0/2718842159 conn(0x7facf80cc120 msgr2=0x7facf80cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 -- 172.21.15.129:0/2718842159 shutdown_connections 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.209+0000 7facfe86b640 1 -- 172.21.15.129:0/2718842159 wait complete. 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.211+0000 7facfe86b640 1 Processor -- start 2024-03-23T15:47:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.211+0000 7facfe86b640 1 -- start start 2024-03-23T15:47:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.211+0000 7facfe86b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf80c6b20 0x7facf8149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.211+0000 7facfe86b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 0x7facf8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facfe86b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf815e540 0x7facf81524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facfe86b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7facf80af6a0 con 0x7facf815e540 2024-03-23T15:47:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facfe86b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facf8152c50 con 0x7facf80c6b20 2024-03-23T15:47:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facfe86b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7facf80cbc90 con 0x7facf815dc20 2024-03-23T15:47:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facf77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 0x7facf8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facf7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf80c6b20 0x7facf8149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facf77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 0x7facf8151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59274/0 (socket says 172.21.15.129:59274) 2024-03-23T15:47:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.212+0000 7facf77fe640 1 -- 172.21.15.129:0/32218328 learned_addr learned my addr 172.21.15.129:0/32218328 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:22.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.214+0000 7facfce08640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf815e540 0x7facf81524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.217+0000 7facf7fff640 1 -- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 msgr2=0x7facf8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.217+0000 7facf7fff640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 0x7facf8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.217+0000 7facf7fff640 1 -- 172.21.15.129:0/32218328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf815e540 msgr2=0x7facf81524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.217+0000 7facf7fff640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf815e540 0x7facf81524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.217+0000 7facf7fff640 1 -- 172.21.15.129:0/32218328 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facec0094d0 con 0x7facf80c6b20 2024-03-23T15:47:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.217+0000 7facf77fe640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 0x7facf8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:47:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.220+0000 7facf7fff640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf80c6b20 0x7facf8149d60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facec033000 tx=0x7facec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:22.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.223+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec039e20 con 0x7facf80c6b20 2024-03-23T15:47:22.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.223+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec043400 con 0x7facf80c6b20 2024-03-23T15:47:22.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.223+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec0437b0 con 0x7facf80c6b20 2024-03-23T15:47:22.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.223+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf815bbd0 con 0x7facf80c6b20 2024-03-23T15:47:22.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.223+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf80af050 con 0x7facf80c6b20 2024-03-23T15:47:22.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.224+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf80af850 con 0x7facf80c6b20 2024-03-23T15:47:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.467+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7facec043910 con 0x7facf80c6b20 2024-03-23T15:47:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.467+0000 7facf4ff9640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7face406b890 0x7face406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.467+0000 7facf77fe640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7face406b890 0x7face406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.468+0000 7facf77fe640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7face406b890 0x7face406dd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7facf815b340 tx=0x7face80090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:22.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.470+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7facec03e070 con 0x7facf80c6b20 2024-03-23T15:47:22.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.480+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7facec078870 con 0x7facf80c6b20 2024-03-23T15:47:22.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.663+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7facf8075740 con 0x7facf80c6b20 2024-03-23T15:47:22.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.870+0000 7facf4ff9640 1 -- 172.21.15.129:0/32218328 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12920 (secure 0 0 0) 0x7facec034090 con 0x7facf80c6b20 2024-03-23T15:47:22.873 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:22.873 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:15.536738+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:22.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7face406b890 msgr2=0x7face406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7face406b890 0x7face406dd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7facf815b340 tx=0x7face80090e0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf80c6b20 msgr2=0x7facf8149d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf80c6b20 0x7facf8149d60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facec033000 tx=0x7facec004730 comp rx=0 tx=0).stop 2024-03-23T15:47:22.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 shutdown_connections 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf815e540 0x7facf81524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facf815dc20 0x7facf8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7face406b890 0x7face406dd50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 --2- 172.21.15.129:0/32218328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facf80c6b20 0x7facf8149d60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 >> 172.21.15.129:0/32218328 conn(0x7facf80cc120 msgr2=0x7facf814e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 shutdown_connections 2024-03-23T15:47:22.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.872+0000 7facfe86b640 1 -- 172.21.15.129:0/32218328 wait complete. 2024-03-23T15:47:22.886 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T15:47:11.061293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T15:47:22.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T15:47:22.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.987+0000 7f0e9df93640 1 Processor -- start 2024-03-23T15:47:22.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.987+0000 7f0e9df93640 1 -- start start 2024-03-23T15:47:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.988+0000 7f0e9df93640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0e980715c0 0x7f0e9815a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.988+0000 7f0e9df93640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e980c2d00 con 0x7f0e98140390 2024-03-23T15:47:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.988+0000 7f0e9df93640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e980c35a0 con 0x7f0e9815aad0 2024-03-23T15:47:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.988+0000 7f0e9df93640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f0e980c3a30 con 0x7f0e980715c0 2024-03-23T15:47:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.989+0000 7f0e96ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0e980715c0 0x7f0e9815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.989+0000 7f0e97fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0e9815aad0 0x7f0e98071a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55908/0 (socket says 172.21.15.129:55908) 2024-03-23T15:47:22.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.989+0000 7f0e97fff640 1 -- 172.21.15.129:0/3867416658 learned_addr learned my addr 172.21.15.129:0/3867416658 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:22.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.991+0000 7f0e967fc640 1 -- 172.21.15.129:0/3867416658 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954803356 0 0) 0x7f0e980c35a0 con 0x7f0e9815aad0 2024-03-23T15:47:22.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.991+0000 7f0e967fc640 1 -- 172.21.15.129:0/3867416658 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e80003520 con 0x7f0e9815aad0 2024-03-23T15:47:22.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.992+0000 7f0e96ffd640 1 -- 172.21.15.129:0/3867416658 >> v1:172.21.15.157:6789/0 conn(0x7f0e9815aad0 legacy=0x7f0e98071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.992+0000 7f0e96ffd640 1 -- 172.21.15.129:0/3867416658 >> v1:172.21.15.129:6789/0 conn(0x7f0e98140390 legacy=0x7f0e980c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:22.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.992+0000 7f0e96ffd640 1 -- 172.21.15.129:0/3867416658 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e980c58b0 con 0x7f0e980715c0 2024-03-23T15:47:22.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.994+0000 7f0e96ffd640 1 --2- 172.21.15.129:0/3867416658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e980715c0 0x7f0e9815a590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89f5695676158e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:23.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.998+0000 7f0e967fc640 1 -- 172.21.15.129:0/3867416658 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c030610 con 0x7f0e980715c0 2024-03-23T15:47:23.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.998+0000 7f0e967fc640 1 -- 172.21.15.129:0/3867416658 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e8c030770 con 0x7f0e980715c0 2024-03-23T15:47:23.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.998+0000 7f0e967fc640 1 -- 172.21.15.129:0/3867416658 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c030b00 con 0x7f0e980715c0 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 -- 172.21.15.129:0/3867416658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e980715c0 msgr2=0x7f0e9815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 --2- 172.21.15.129:0/3867416658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e980715c0 0x7f0e9815a590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c02b960 comp rx=0 tx=0).stop 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 -- 172.21.15.129:0/3867416658 shutdown_connections 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 --2- 172.21.15.129:0/3867416658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e980715c0 0x7f0e9815a590 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 -- 172.21.15.129:0/3867416658 >> 172.21.15.129:0/3867416658 conn(0x7f0e980cb800 msgr2=0x7f0e980cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 -- 172.21.15.129:0/3867416658 shutdown_connections 2024-03-23T15:47:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:22.999+0000 7f0e9df93640 1 -- 172.21.15.129:0/3867416658 wait complete. 2024-03-23T15:47:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.000+0000 7f0e9df93640 1 Processor -- start 2024-03-23T15:47:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.000+0000 7f0e9df93640 1 -- start start 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e9df93640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e980715c0 0x7f0e9813cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e9df93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 0x7f0e981f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e9df93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e9815aad0 0x7f0e9813e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e9df93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9813b380 con 0x7f0e98140390 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e9df93640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9813f7d0 con 0x7f0e980715c0 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e9df93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9813d970 con 0x7f0e9815aad0 2024-03-23T15:47:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.001+0000 7f0e96ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 0x7f0e981f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.002+0000 7f0e97fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e9815aad0 0x7f0e9813e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.002+0000 7f0e977fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e980715c0 0x7f0e9813cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.002+0000 7f0e96ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 0x7f0e981f1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37428/0 (socket says 172.21.15.129:37428) 2024-03-23T15:47:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.002+0000 7f0e96ffd640 1 -- 172.21.15.129:0/2027277062 learned_addr learned my addr 172.21.15.129:0/2027277062 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:23.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.005+0000 7f0e96ffd640 1 -- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e9815aad0 msgr2=0x7f0e9813e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.005+0000 7f0e96ffd640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e9815aad0 0x7f0e9813e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.005+0000 7f0e96ffd640 1 -- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e980715c0 msgr2=0x7f0e9813cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.005+0000 7f0e96ffd640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e980715c0 0x7f0e9813cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.005+0000 7f0e96ffd640 1 -- 172.21.15.129:0/2027277062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c009400 con 0x7f0e98140390 2024-03-23T15:47:23.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.007+0000 7f0e96ffd640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 0x7f0e981f1750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009890 tx=0x7f0e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:23.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.010+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c03e070 con 0x7f0e98140390 2024-03-23T15:47:23.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.011+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e8c0397a0 con 0x7f0e98140390 2024-03-23T15:47:23.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.011+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9813dbf0 con 0x7f0e98140390 2024-03-23T15:47:23.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.011+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c039b70 con 0x7f0e98140390 2024-03-23T15:47:23.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.011+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e981f22f0 con 0x7f0e98140390 2024-03-23T15:47:23.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.012+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9813b530 con 0x7f0e98140390 2024-03-23T15:47:23.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.252+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0e8c039cd0 con 0x7f0e98140390 2024-03-23T15:47:23.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.252+0000 7f0e7ffff640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8406b860 0x7f0e8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.253+0000 7f0e977fe640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8406b860 0x7f0e8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.253+0000 7f0e977fe640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8406b860 0x7f0e8406dd20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e88001780 tx=0x7f0e88009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:23.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.267+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0e8c07a800 con 0x7f0e98140390 2024-03-23T15:47:23.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.267+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0e8c076aa0 con 0x7f0e98140390 2024-03-23T15:47:23.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.441+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0e98075740 con 0x7f0e98140390 2024-03-23T15:47:23.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.461+0000 7f0e7ffff640 1 -- 172.21.15.129:0/2027277062 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f0e8c048020 con 0x7f0e98140390 2024-03-23T15:47:23.464 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8406b860 msgr2=0x7f0e8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8406b860 0x7f0e8406dd20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e88001780 tx=0x7f0e88009040 comp rx=0 tx=0).stop 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 msgr2=0x7f0e981f1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 0x7f0e981f1750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009890 tx=0x7f0e8c0046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 shutdown_connections 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e9815aad0 0x7f0e9813e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e98140390 0x7f0e981f1750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8406b860 0x7f0e8406dd20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 --2- 172.21.15.129:0/2027277062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e980715c0 0x7f0e9813cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 >> 172.21.15.129:0/2027277062 conn(0x7f0e980cb800 msgr2=0x7f0e9806ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 shutdown_connections 2024-03-23T15:47:23.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.464+0000 7f0e9df93640 1 -- 172.21.15.129:0/2027277062 wait complete. 2024-03-23T15:47:23.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T15:47:23.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.576+0000 7f5fafc0b640 1 Processor -- start 2024-03-23T15:47:23.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.576+0000 7f5fafc0b640 1 -- start start 2024-03-23T15:47:23.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.577+0000 7f5fafc0b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5fa815d830 0x7f5fa80cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.577+0000 7f5fafc0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80cf190 0x7f5fa815e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.577+0000 7f5fafc0b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa80cda60 con 0x7f5fa80caf30 2024-03-23T15:47:23.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.577+0000 7f5fafc0b640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa806bfd0 con 0x7f5fa815d830 2024-03-23T15:47:23.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.577+0000 7f5fafc0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa806c2b0 con 0x7f5fa80cf190 2024-03-23T15:47:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.578+0000 7f5fad9a7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5fa815d830 0x7f5fa80cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.578+0000 7f5fae1a8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80cf190 0x7f5fa815e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.578+0000 7f5fad1a6640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5fa80caf30 0x7f5fa815dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55924/0 (socket says 172.21.15.129:55924) 2024-03-23T15:47:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.578+0000 7f5fad1a6640 1 -- 172.21.15.129:0/2578219608 learned_addr learned my addr 172.21.15.129:0/2578219608 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.580+0000 7f5fad9a7640 1 -- 172.21.15.129:0/2578219608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80cf190 msgr2=0x7f5fa815e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.580+0000 7f5fad9a7640 1 --2- 172.21.15.129:0/2578219608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80cf190 0x7f5fa815e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.580+0000 7f5fad9a7640 1 -- 172.21.15.129:0/2578219608 >> v1:172.21.15.157:6789/0 conn(0x7f5fa80caf30 legacy=0x7f5fa815dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.580+0000 7f5fad9a7640 1 -- 172.21.15.129:0/2578219608 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa806c590 con 0x7f5fa815d830 2024-03-23T15:47:23.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.582+0000 7f5fad9a7640 1 --2- 172.21.15.129:0/2578219608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa815d830 0x7f5fa80cdef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40097a0 tx=0x7f5fa402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77a74efb3406d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:23.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.586+0000 7f5fac9a5640 1 -- 172.21.15.129:0/2578219608 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4030610 con 0x7f5fa815d830 2024-03-23T15:47:23.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.586+0000 7f5fac9a5640 1 -- 172.21.15.129:0/2578219608 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa4030770 con 0x7f5fa815d830 2024-03-23T15:47:23.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.586+0000 7f5fac9a5640 1 -- 172.21.15.129:0/2578219608 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4030b00 con 0x7f5fa815d830 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2578219608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa815d830 msgr2=0x7f5fa80cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2578219608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa815d830 0x7f5fa80cdef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40097a0 tx=0x7f5fa402b960 comp rx=0 tx=0).stop 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2578219608 shutdown_connections 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2578219608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80cf190 0x7f5fa815e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2578219608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa815d830 0x7f5fa80cdef0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2578219608 >> 172.21.15.129:0/2578219608 conn(0x7f5fa80b16b0 msgr2=0x7f5fa80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2578219608 shutdown_connections 2024-03-23T15:47:23.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.587+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2578219608 wait complete. 2024-03-23T15:47:23.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.588+0000 7f5fafc0b640 1 Processor -- start 2024-03-23T15:47:23.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.588+0000 7f5fafc0b640 1 -- start start 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fafc0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80caf30 0x7f5fa81511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fafc0b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 0x7f5fa80cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fafc0b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fa815d830 0x7f5fa80cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fafc0b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa80c9390 con 0x7f5fa80cf190 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fafc0b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa80c4dc0 con 0x7f5fa815d830 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fafc0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa80c95a0 con 0x7f5fa80caf30 2024-03-23T15:47:23.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.589+0000 7f5fad1a6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 0x7f5fa80cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.590+0000 7f5fae1a8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fa815d830 0x7f5fa80cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.590+0000 7f5fad9a7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80caf30 0x7f5fa81511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.590+0000 7f5fad1a6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 0x7f5fa80cb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37438/0 (socket says 172.21.15.129:37438) 2024-03-23T15:47:23.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.590+0000 7f5fad1a6640 1 -- 172.21.15.129:0/2846653434 learned_addr learned my addr 172.21.15.129:0/2846653434 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:23.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.593+0000 7f5fad9a7640 1 -- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fa815d830 msgr2=0x7f5fa80cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.593+0000 7f5fad9a7640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fa815d830 0x7f5fa80cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.593+0000 7f5fad9a7640 1 -- 172.21.15.129:0/2846653434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 msgr2=0x7f5fa80cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:23.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.593+0000 7f5fad9a7640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 0x7f5fa80cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:23.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.593+0000 7f5fad9a7640 1 -- 172.21.15.129:0/2846653434 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa4009400 con 0x7f5fa80caf30 2024-03-23T15:47:23.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.593+0000 7f5fad1a6640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 0x7f5fa80cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:47:23.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.595+0000 7f5fad9a7640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80caf30 0x7f5fa81511f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4033000 tx=0x7f5fa40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:23.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.599+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa403e070 con 0x7f5fa80caf30 2024-03-23T15:47:23.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.599+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa4030e10 con 0x7f5fa80caf30 2024-03-23T15:47:23.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.599+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4039c30 con 0x7f5fa80caf30 2024-03-23T15:47:23.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.599+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa8156d00 con 0x7f5fa80caf30 2024-03-23T15:47:23.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.600+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa8151730 con 0x7f5fa80caf30 2024-03-23T15:47:23.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.600+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa80afb30 con 0x7f5fa80caf30 2024-03-23T15:47:23.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.619+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5fa404a3f0 con 0x7f5fa80caf30 2024-03-23T15:47:23.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.619+0000 7f5f9e7fc640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5f8c06b890 0x7f5f8c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:23.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.620+0000 7f5fad1a6640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5f8c06b890 0x7f5f8c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:23.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.620+0000 7f5fad1a6640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5f8c06b890 0x7f5f8c06dd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80b1c70 tx=0x7f5f98009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:23.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.849+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f5fa407a700 con 0x7f5fa80caf30 2024-03-23T15:47:23.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:23.860+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5fa40769a0 con 0x7f5fa80caf30 2024-03-23T15:47:24.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.040+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5fa80681d0 con 0x7f5fa80caf30 2024-03-23T15:47:24.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.046+0000 7f5f9e7fc640 1 -- 172.21.15.129:0/2846653434 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f5fa407a060 con 0x7f5fa80caf30 2024-03-23T15:47:24.049 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-03-23T15:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-03-23T15:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-03-23T15:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-03-23T15:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-03-23T15:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-03-23T15:47:24.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5f8c06b890 msgr2=0x7f5f8c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5f8c06b890 0x7f5f8c06dd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80b1c70 tx=0x7f5f98009040 comp rx=0 tx=0).stop 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80caf30 msgr2=0x7f5fa81511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80caf30 0x7f5fa81511f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4033000 tx=0x7f5fa40046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 shutdown_connections 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5f8c06b890 0x7f5f8c06dd50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fa815d830 0x7f5fa80cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fa80cf190 0x7f5fa80cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 --2- 172.21.15.129:0/2846653434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fa80caf30 0x7f5fa81511f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 >> 172.21.15.129:0/2846653434 conn(0x7f5fa80b16b0 msgr2=0x7f5fa815b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 shutdown_connections 2024-03-23T15:47:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.048+0000 7f5fafc0b640 1 -- 172.21.15.129:0/2846653434 wait complete. 2024-03-23T15:47:24.062 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T15:47:24.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T15:47:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.164+0000 7f978d833640 1 Processor -- start 2024-03-23T15:47:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.164+0000 7f978d833640 1 -- start start 2024-03-23T15:47:24.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.165+0000 7f978d833640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788152330 0x7f9788158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:24.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.165+0000 7f978d833640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f978815c500 con 0x7f9788071d60 2024-03-23T15:47:24.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.165+0000 7f978d833640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9788158f80 con 0x7f9788072560 2024-03-23T15:47:24.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.165+0000 7f978d833640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f978815c0e0 con 0x7f9788152330 2024-03-23T15:47:24.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.166+0000 7f9786ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788152330 0x7f9788158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:24.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.166+0000 7f97867fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9788071d60 0x7f978806b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60884/0 (socket says 172.21.15.129:60884) 2024-03-23T15:47:24.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.166+0000 7f97867fc640 1 -- 172.21.15.129:0/1272733009 learned_addr learned my addr 172.21.15.129:0/1272733009 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:24.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.168+0000 7f9785ffb640 1 -- 172.21.15.129:0/1272733009 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262098532 0 0) 0x7f978815c500 con 0x7f9788071d60 2024-03-23T15:47:24.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.168+0000 7f9785ffb640 1 -- 172.21.15.129:0/1272733009 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9770003520 con 0x7f9788071d60 2024-03-23T15:47:24.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.169+0000 7f9786ffd640 1 -- 172.21.15.129:0/1272733009 >> v1:172.21.15.157:6789/0 conn(0x7f9788072560 legacy=0x7f9788072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.169+0000 7f9786ffd640 1 -- 172.21.15.129:0/1272733009 >> v1:172.21.15.129:6789/0 conn(0x7f9788071d60 legacy=0x7f978806b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.169+0000 7f9786ffd640 1 -- 172.21.15.129:0/1272733009 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9788073530 con 0x7f9788152330 2024-03-23T15:47:24.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.171+0000 7f9786ffd640 1 --2- 172.21.15.129:0/1272733009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788152330 0x7f9788158a40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9778009dc0 tx=0x7f977802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69336829372acb01 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:24.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.173+0000 7f9785ffb640 1 -- 172.21.15.129:0/1272733009 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977802ccd0 con 0x7f9788152330 2024-03-23T15:47:24.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f9785ffb640 1 -- 172.21.15.129:0/1272733009 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977802ce30 con 0x7f9788152330 2024-03-23T15:47:24.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f9785ffb640 1 -- 172.21.15.129:0/1272733009 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778034640 con 0x7f9788152330 2024-03-23T15:47:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f978d833640 1 -- 172.21.15.129:0/1272733009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788152330 msgr2=0x7f9788158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f978d833640 1 --2- 172.21.15.129:0/1272733009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788152330 0x7f9788158a40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9778009dc0 tx=0x7f977802bf20 comp rx=0 tx=0).stop 2024-03-23T15:47:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f978d833640 1 -- 172.21.15.129:0/1272733009 shutdown_connections 2024-03-23T15:47:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f978d833640 1 --2- 172.21.15.129:0/1272733009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788152330 0x7f9788158a40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.175+0000 7f978d833640 1 -- 172.21.15.129:0/1272733009 >> 172.21.15.129:0/1272733009 conn(0x7f97880c8bb0 msgr2=0x7f97880c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:24.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.176+0000 7f978d833640 1 -- 172.21.15.129:0/1272733009 shutdown_connections 2024-03-23T15:47:24.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.176+0000 7f978d833640 1 -- 172.21.15.129:0/1272733009 wait complete. 2024-03-23T15:47:24.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 Processor -- start 2024-03-23T15:47:24.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 -- start start 2024-03-23T15:47:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 0x7f97880c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788072560 0x7f97881e47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9788152330 0x7f9788159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9788159170 con 0x7f9788071d60 2024-03-23T15:47:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9788159a10 con 0x7f9788152330 2024-03-23T15:47:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.177+0000 7f978d833640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f978814f680 con 0x7f9788072560 2024-03-23T15:47:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.178+0000 7f9786ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 0x7f97880c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.178+0000 7f97867fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788072560 0x7f97881e47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.178+0000 7f97877fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9788152330 0x7f9788159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.178+0000 7f9786ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 0x7f97880c6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37452/0 (socket says 172.21.15.129:37452) 2024-03-23T15:47:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.178+0000 7f9786ffd640 1 -- 172.21.15.129:0/1621832513 learned_addr learned my addr 172.21.15.129:0/1621832513 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:24.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.182+0000 7f9786ffd640 1 -- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788072560 msgr2=0x7f97881e47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.182+0000 7f9786ffd640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788072560 0x7f97881e47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.182+0000 7f9786ffd640 1 -- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9788152330 msgr2=0x7f9788159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:24.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.182+0000 7f9786ffd640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9788152330 0x7f9788159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:24.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.182+0000 7f9786ffd640 1 -- 172.21.15.129:0/1621832513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9778009a70 con 0x7f9788071d60 2024-03-23T15:47:24.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.184+0000 7f9786ffd640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 0x7f97880c6190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f977802dfd0 tx=0x7f9778004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:24.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.187+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778004880 con 0x7f9788071d60 2024-03-23T15:47:24.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.187+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9778009450 con 0x7f9788071d60 2024-03-23T15:47:24.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.187+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97881ee8a0 con 0x7f9788071d60 2024-03-23T15:47:24.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.187+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778040670 con 0x7f9788071d60 2024-03-23T15:47:24.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.187+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97881593c0 con 0x7f9788071d60 2024-03-23T15:47:24.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.188+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f978806b360 con 0x7f9788071d60 2024-03-23T15:47:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.433+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97780407d0 con 0x7f9788071d60 2024-03-23T15:47:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.434+0000 7f976ffff640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f977406b840 0x7f977406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.434+0000 7f97867fc640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f977406b840 0x7f977406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.434+0000 7f97867fc640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f977406b840 0x7f977406dd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9788151240 tx=0x7f977c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:24.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.458+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f977807b9c0 con 0x7f9788071d60 2024-03-23T15:47:24.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.458+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9778077c60 con 0x7f9788071d60 2024-03-23T15:47:24.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:24.630+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f97880cb550 con 0x7f9788071d60 2024-03-23T15:47:25.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.018+0000 7f976ffff640 1 -- 172.21.15.129:0/1621832513 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f97780390b0 con 0x7f9788071d60 2024-03-23T15:47:25.022 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.020+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f977406b840 msgr2=0x7f977406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.020+0000 7f978d833640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f977406b840 0x7f977406dd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9788151240 tx=0x7f977c007d30 comp rx=0 tx=0).stop 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 msgr2=0x7f97880c6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 0x7f97880c6190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f977802dfd0 tx=0x7f9778004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 shutdown_connections 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f977406b840 0x7f977406dd00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9788152330 0x7f9788159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9788072560 0x7f97881e47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 --2- 172.21.15.129:0/1621832513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9788071d60 0x7f97880c6190 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 >> 172.21.15.129:0/1621832513 conn(0x7f97880c8bb0 msgr2=0x7f97881508f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:25.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 shutdown_connections 2024-03-23T15:47:25.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.021+0000 7f978d833640 1 -- 172.21.15.129:0/1621832513 wait complete. 2024-03-23T15:47:25.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T15:47:25.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.143+0000 7f698a92e640 1 Processor -- start 2024-03-23T15:47:25.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.143+0000 7f698a92e640 1 -- start start 2024-03-23T15:47:25.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f698a92e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f698415e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:25.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f698a92e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f698406bf30 0x7f698406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:25.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f698a92e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69841549d0 con 0x7f69840c8510 2024-03-23T15:47:25.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f698a92e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f698406bae0 con 0x7f698406bf30 2024-03-23T15:47:25.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f698a92e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f698406c900 con 0x7f69841545f0 2024-03-23T15:47:25.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f6988ecb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f698406bf30 0x7f698406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:25.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.144+0000 7f6983fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f698415e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:25.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.145+0000 7f6983fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f698415e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59336/0 (socket says 172.21.15.129:59336) 2024-03-23T15:47:25.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.145+0000 7f6983fff640 1 -- 172.21.15.129:0/1911364580 learned_addr learned my addr 172.21.15.129:0/1911364580 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:25.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.147+0000 7f6983fff640 1 -- 172.21.15.129:0/1911364580 >> v2:172.21.15.129:3300/0 conn(0x7f698406bf30 msgr2=0x7f698406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:25.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.147+0000 7f6983fff640 1 --2- 172.21.15.129:0/1911364580 >> v2:172.21.15.129:3300/0 conn(0x7f698406bf30 0x7f698406c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.147+0000 7f6983fff640 1 -- 172.21.15.129:0/1911364580 >> v1:172.21.15.157:6789/0 conn(0x7f69840c8510 legacy=0x7f69840c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:25.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.147+0000 7f6983fff640 1 -- 172.21.15.129:0/1911364580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698415e700 con 0x7f69841545f0 2024-03-23T15:47:25.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.149+0000 7f6983fff640 1 --2- 172.21.15.129:0/1911364580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f698415e1c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6974009870 tx=0x7f697402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e121af57508d28e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:25.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.153+0000 7f6982ffd640 1 -- 172.21.15.129:0/1911364580 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974030840 con 0x7f69841545f0 2024-03-23T15:47:25.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.153+0000 7f6982ffd640 1 -- 172.21.15.129:0/1911364580 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69740309a0 con 0x7f69841545f0 2024-03-23T15:47:25.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.153+0000 7f6982ffd640 1 -- 172.21.15.129:0/1911364580 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974030d30 con 0x7f69841545f0 2024-03-23T15:47:25.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.153+0000 7f698a92e640 1 -- 172.21.15.129:0/1911364580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 msgr2=0x7f698415e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.153+0000 7f698a92e640 1 --2- 172.21.15.129:0/1911364580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f698415e1c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6974009870 tx=0x7f697402b980 comp rx=0 tx=0).stop 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.154+0000 7f698a92e640 1 -- 172.21.15.129:0/1911364580 shutdown_connections 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.154+0000 7f698a92e640 1 --2- 172.21.15.129:0/1911364580 >> v2:172.21.15.129:3300/0 conn(0x7f698406bf30 0x7f698406c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.154+0000 7f698a92e640 1 --2- 172.21.15.129:0/1911364580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f698415e1c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.154+0000 7f698a92e640 1 -- 172.21.15.129:0/1911364580 >> 172.21.15.129:0/1911364580 conn(0x7f69840cc230 msgr2=0x7f69840cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.154+0000 7f698a92e640 1 -- 172.21.15.129:0/1911364580 shutdown_connections 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.154+0000 7f698a92e640 1 -- 172.21.15.129:0/1911364580 wait complete. 2024-03-23T15:47:25.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.155+0000 7f698a92e640 1 Processor -- start 2024-03-23T15:47:25.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.155+0000 7f698a92e640 1 -- start start 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f698a92e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698406bf30 0x7f69841e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f698a92e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69840c8510 0x7f69841e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f698a92e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f69840af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f698a92e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69841e3fc0 con 0x7f698406bf30 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f698a92e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f69841e5300 con 0x7f69840c8510 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f698a92e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f698406ea10 con 0x7f69841545f0 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f6988ecb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f69840af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f6983fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698406bf30 0x7f69841e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f6988ecb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f69840af420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59348/0 (socket says 172.21.15.129:59348) 2024-03-23T15:47:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.156+0000 7f6988ecb640 1 -- 172.21.15.129:0/1604215689 learned_addr learned my addr 172.21.15.129:0/1604215689 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:25.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.160+0000 7f6983fff640 1 -- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 msgr2=0x7f69840af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:25.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.160+0000 7f6983fff640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f69840af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.160+0000 7f6983fff640 1 -- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69840c8510 msgr2=0x7f69841e4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:25.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.160+0000 7f6983fff640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69840c8510 0x7f69841e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:25.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.160+0000 7f6983fff640 1 -- 172.21.15.129:0/1604215689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69740094d0 con 0x7f698406bf30 2024-03-23T15:47:25.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.162+0000 7f6983fff640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698406bf30 0x7f69841e5a10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6974033000 tx=0x7f6974004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:25.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.179+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974039a90 con 0x7f698406bf30 2024-03-23T15:47:25.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.179+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6974039bf0 con 0x7f698406bf30 2024-03-23T15:47:25.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.179+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69841e60a0 con 0x7f698406bf30 2024-03-23T15:47:25.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.179+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69740433f0 con 0x7f698406bf30 2024-03-23T15:47:25.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.179+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69841e3690 con 0x7f698406bf30 2024-03-23T15:47:25.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.180+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69840c6e80 con 0x7f698406bf30 2024-03-23T15:47:25.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.198+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f697404c3b0 con 0x7f698406bf30 2024-03-23T15:47:25.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.198+0000 7f6980ff9640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f697006b890 0x7f697006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:25.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.199+0000 7f69837fe640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f697006b890 0x7f697006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:25.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.199+0000 7f69837fe640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f697006b890 0x7f697006dd50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f698414c1d0 tx=0x7f6978007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:25.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.437+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f697407ae10 con 0x7f698406bf30 2024-03-23T15:47:25.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.437+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f69740770b0 con 0x7f698406bf30 2024-03-23T15:47:25.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:25.610+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f69841549d0 con 0x7f698406bf30 2024-03-23T15:47:26.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.112+0000 7f6980ff9640 1 -- 172.21.15.129:0/1604215689 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f69841549d0 con 0x7f698406bf30 2024-03-23T15:47:26.115 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-03-23T15:47:26.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f697006b890 msgr2=0x7f697006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f697006b890 0x7f697006dd50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f698414c1d0 tx=0x7f6978007990 comp rx=0 tx=0).stop 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698406bf30 msgr2=0x7f69841e5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698406bf30 0x7f69841e5a10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6974033000 tx=0x7f6974004730 comp rx=0 tx=0).stop 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 shutdown_connections 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69841545f0 0x7f69840af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f697006b890 0x7f697006dd50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69840c8510 0x7f69841e4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 --2- 172.21.15.129:0/1604215689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698406bf30 0x7f69841e5a10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 >> 172.21.15.129:0/1604215689 conn(0x7f69840cc230 msgr2=0x7f6984151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 shutdown_connections 2024-03-23T15:47:26.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.115+0000 7f698a92e640 1 -- 172.21.15.129:0/1604215689 wait complete. 2024-03-23T15:47:26.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T15:47:26.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.240+0000 7fb175447640 1 Processor -- start 2024-03-23T15:47:26.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.240+0000 7fb175447640 1 -- start start 2024-03-23T15:47:26.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.241+0000 7fb175447640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb17006b780 0x7fb170153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:26.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.241+0000 7fb175447640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1700733e0 0x7fb170154500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:26.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.241+0000 7fb175447640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb170073170 con 0x7fb17006b3a0 2024-03-23T15:47:26.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.241+0000 7fb175447640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb17015e7c0 con 0x7fb17006b780 2024-03-23T15:47:26.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.241+0000 7fb175447640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb17015d8a0 con 0x7fb1700733e0 2024-03-23T15:47:26.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.242+0000 7fb167fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb17006b780 0x7fb170153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:26.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.242+0000 7fb167fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb17006b780 0x7fb170153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37480/0 (socket says 172.21.15.129:37480) 2024-03-23T15:47:26.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.242+0000 7fb167fff640 1 -- 172.21.15.129:0/1993638230 learned_addr learned my addr 172.21.15.129:0/1993638230 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:26.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.244+0000 7fb16f7fe640 1 --2- 172.21.15.129:0/1993638230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1700733e0 0x7fb170154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:26.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.245+0000 7fb167fff640 1 -- 172.21.15.129:0/1993638230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1700733e0 msgr2=0x7fb170154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:26.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.245+0000 7fb167fff640 1 --2- 172.21.15.129:0/1993638230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1700733e0 0x7fb170154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.245+0000 7fb167fff640 1 -- 172.21.15.129:0/1993638230 >> v1:172.21.15.157:6789/0 conn(0x7fb17006b3a0 legacy=0x7fb1700c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.245+0000 7fb167fff640 1 -- 172.21.15.129:0/1993638230 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17015db40 con 0x7fb17006b780 2024-03-23T15:47:26.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.247+0000 7fb167fff640 1 --2- 172.21.15.129:0/1993638230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17006b780 0x7fb170153fc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb1580098a0 tx=0x7fb15802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f71fb5eca959e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:26.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb16e7fc640 1 -- 172.21.15.129:0/1993638230 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158031640 con 0x7fb17006b780 2024-03-23T15:47:26.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb16e7fc640 1 -- 172.21.15.129:0/1993638230 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1580317a0 con 0x7fb17006b780 2024-03-23T15:47:26.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb16e7fc640 1 -- 172.21.15.129:0/1993638230 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158031b30 con 0x7fb17006b780 2024-03-23T15:47:26.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb175447640 1 -- 172.21.15.129:0/1993638230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17006b780 msgr2=0x7fb170153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:26.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb175447640 1 --2- 172.21.15.129:0/1993638230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17006b780 0x7fb170153fc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb1580098a0 tx=0x7fb15802b8f0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb175447640 1 -- 172.21.15.129:0/1993638230 shutdown_connections 2024-03-23T15:47:26.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb175447640 1 --2- 172.21.15.129:0/1993638230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1700733e0 0x7fb170154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb175447640 1 --2- 172.21.15.129:0/1993638230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb17006b780 0x7fb170153fc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.258+0000 7fb175447640 1 -- 172.21.15.129:0/1993638230 >> 172.21.15.129:0/1993638230 conn(0x7fb170062180 msgr2=0x7fb170070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.259+0000 7fb175447640 1 -- 172.21.15.129:0/1993638230 shutdown_connections 2024-03-23T15:47:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.259+0000 7fb175447640 1 -- 172.21.15.129:0/1993638230 wait complete. 2024-03-23T15:47:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 Processor -- start 2024-03-23T15:47:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 -- start start 2024-03-23T15:47:26.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 0x7fb1701476c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:26.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17006b780 0x7fb1700cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:26.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1700733e0 0x7fb1701eff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:26.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb170147230 con 0x7fb1700733e0 2024-03-23T15:47:26.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb1700c6af0 con 0x7fb17006b780 2024-03-23T15:47:26.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.260+0000 7fb175447640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb1700cada0 con 0x7fb17006b3a0 2024-03-23T15:47:26.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.261+0000 7fb16effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 0x7fb1701476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:26.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.261+0000 7fb16effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 0x7fb1701476c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59370/0 (socket says 172.21.15.129:59370) 2024-03-23T15:47:26.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.261+0000 7fb16effd640 1 -- 172.21.15.129:0/3978578095 learned_addr learned my addr 172.21.15.129:0/3978578095 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:26.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.265+0000 7fb16effd640 1 -- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17006b780 msgr2=0x7fb1700cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:26.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.265+0000 7fb16effd640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17006b780 0x7fb1700cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.265+0000 7fb16effd640 1 -- 172.21.15.129:0/3978578095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1700733e0 msgr2=0x7fb1701eff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:26.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.265+0000 7fb16effd640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1700733e0 0x7fb1701eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:26.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.265+0000 7fb16effd640 1 -- 172.21.15.129:0/3978578095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1580094d0 con 0x7fb17006b3a0 2024-03-23T15:47:26.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.268+0000 7fb16effd640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 0x7fb1701476c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c410 tx=0x7fb16000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:26.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.272+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1600035f0 con 0x7fb17006b3a0 2024-03-23T15:47:26.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.272+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb160003750 con 0x7fb17006b3a0 2024-03-23T15:47:26.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.272+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1700cb060 con 0x7fb17006b3a0 2024-03-23T15:47:26.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.273+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160003b40 con 0x7fb17006b3a0 2024-03-23T15:47:26.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.273+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb170146ec0 con 0x7fb17006b3a0 2024-03-23T15:47:26.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.273+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1700cb580 con 0x7fb17006b3a0 2024-03-23T15:47:26.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.525+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb160003ca0 con 0x7fb17006b3a0 2024-03-23T15:47:26.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.526+0000 7fb1677fe640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb15006b860 0x7fb15006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:26.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.526+0000 7fb167fff640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb15006b860 0x7fb15006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:26.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.526+0000 7fb167fff640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb15006b860 0x7fb15006dd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb158036000 tx=0x7fb158030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:26.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.529+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fb160058a80 con 0x7fb17006b3a0 2024-03-23T15:47:26.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.540+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb160054620 con 0x7fb17006b3a0 2024-03-23T15:47:26.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:26.772+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb1700cb580 con 0x7fb17006b3a0 2024-03-23T15:47:27.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.640+0000 7fb1677fe640 1 -- 172.21.15.129:0/3978578095 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fb1700cb580 con 0x7fb17006b3a0 2024-03-23T15:47:27.643 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 2 and data pool 3 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.642+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb15006b860 msgr2=0x7fb15006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.642+0000 7fb175447640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb15006b860 0x7fb15006dd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb158036000 tx=0x7fb158030000 comp rx=0 tx=0).stop 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 msgr2=0x7fb1701476c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 0x7fb1701476c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c410 tx=0x7fb16000c8d0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 shutdown_connections 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1700733e0 0x7fb1701eff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb15006b860 0x7fb15006dd20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb17006b780 0x7fb1700cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 --2- 172.21.15.129:0/3978578095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb17006b3a0 0x7fb1701476c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 >> 172.21.15.129:0/3978578095 conn(0x7fb170062180 msgr2=0x7fb170072440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 shutdown_connections 2024-03-23T15:47:27.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.643+0000 7fb175447640 1 -- 172.21.15.129:0/3978578095 wait complete. 2024-03-23T15:47:27.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:27.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.769+0000 7ff77aae6640 1 Processor -- start 2024-03-23T15:47:27.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.769+0000 7ff77aae6640 1 -- start start 2024-03-23T15:47:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.770+0000 7ff77aae6640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff7741544b0 0x7ff77415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.770+0000 7ff77aae6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff77406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.770+0000 7ff77aae6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7740cbcb0 con 0x7ff77406c0b0 2024-03-23T15:47:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.770+0000 7ff77aae6640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ff7740c5be0 con 0x7ff7741544b0 2024-03-23T15:47:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.770+0000 7ff77aae6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff77415b280 con 0x7ff7740caa00 2024-03-23T15:47:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.771+0000 7ff779083640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff77406c0b0 0x7ff7740c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55964/0 (socket says 172.21.15.129:55964) 2024-03-23T15:47:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.771+0000 7ff773fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff77406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.771+0000 7ff779083640 1 -- 172.21.15.129:0/1282533413 learned_addr learned my addr 172.21.15.129:0/1282533413 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.773+0000 7ff7737fe640 1 -- 172.21.15.129:0/1282533413 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1809229578 0 0) 0x7ff7740cbcb0 con 0x7ff77406c0b0 2024-03-23T15:47:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.773+0000 7ff773fff640 1 -- 172.21.15.129:0/1282533413 >> v2:172.21.15.129:3300/0 conn(0x7ff7741544b0 msgr2=0x7ff77415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.773+0000 7ff773fff640 1 --2- 172.21.15.129:0/1282533413 >> v2:172.21.15.129:3300/0 conn(0x7ff7741544b0 0x7ff77415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.773+0000 7ff773fff640 1 -- 172.21.15.129:0/1282533413 >> v1:172.21.15.157:6789/0 conn(0x7ff77406c0b0 legacy=0x7ff7740c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.773+0000 7ff773fff640 1 -- 172.21.15.129:0/1282533413 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77415a9e0 con 0x7ff7740caa00 2024-03-23T15:47:27.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.789+0000 7ff773fff640 1 --2- 172.21.15.129:0/1282533413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff77406bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff7600097a0 tx=0x7ff76002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a345151b8bc7e11d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:27.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.801+0000 7ff7737fe640 1 -- 172.21.15.129:0/1282533413 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760030610 con 0x7ff7740caa00 2024-03-23T15:47:27.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.801+0000 7ff7737fe640 1 -- 172.21.15.129:0/1282533413 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff760030770 con 0x7ff7740caa00 2024-03-23T15:47:27.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.801+0000 7ff7737fe640 1 -- 172.21.15.129:0/1282533413 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760030b00 con 0x7ff7740caa00 2024-03-23T15:47:27.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.801+0000 7ff77aae6640 1 -- 172.21.15.129:0/1282533413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 msgr2=0x7ff77406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:27.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.801+0000 7ff77aae6640 1 --2- 172.21.15.129:0/1282533413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff77406bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff7600097a0 tx=0x7ff76002b960 comp rx=0 tx=0).stop 2024-03-23T15:47:27.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.802+0000 7ff77aae6640 1 -- 172.21.15.129:0/1282533413 shutdown_connections 2024-03-23T15:47:27.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.802+0000 7ff77aae6640 1 --2- 172.21.15.129:0/1282533413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff77406bb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.802+0000 7ff77aae6640 1 --2- 172.21.15.129:0/1282533413 >> v2:172.21.15.129:3300/0 conn(0x7ff7741544b0 0x7ff77415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.802+0000 7ff77aae6640 1 -- 172.21.15.129:0/1282533413 >> 172.21.15.129:0/1282533413 conn(0x7ff774061b50 msgr2=0x7ff774061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:27.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.802+0000 7ff77aae6640 1 -- 172.21.15.129:0/1282533413 shutdown_connections 2024-03-23T15:47:27.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.802+0000 7ff77aae6640 1 -- 172.21.15.129:0/1282533413 wait complete. 2024-03-23T15:47:27.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.803+0000 7ff77aae6640 1 Processor -- start 2024-03-23T15:47:27.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.803+0000 7ff77aae6640 1 -- start start 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.804+0000 7ff77aae6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff77406c0b0 0x7ff7741560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.804+0000 7ff77aae6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff774154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.804+0000 7ff77aae6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7741544b0 0x7ff774155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.804+0000 7ff77aae6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741566f0 con 0x7ff7741544b0 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.804+0000 7ff77aae6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff774159b50 con 0x7ff77406c0b0 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.804+0000 7ff77aae6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff774159590 con 0x7ff7740caa00 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.805+0000 7ff773fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff774154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.805+0000 7ff778882640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff77406c0b0 0x7ff7741560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:27.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.805+0000 7ff773fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff774154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59392/0 (socket says 172.21.15.129:59392) 2024-03-23T15:47:27.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.805+0000 7ff773fff640 1 -- 172.21.15.129:0/4247151447 learned_addr learned my addr 172.21.15.129:0/4247151447 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:27.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.805+0000 7ff778882640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff77406c0b0 0x7ff7741560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52666/0 (socket says 172.21.15.129:52666) 2024-03-23T15:47:27.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.805+0000 7ff779083640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7741544b0 0x7ff774155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:27.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.808+0000 7ff773fff640 1 -- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff77406c0b0 msgr2=0x7ff7741560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:27.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.808+0000 7ff773fff640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff77406c0b0 0x7ff7741560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.808+0000 7ff773fff640 1 -- 172.21.15.129:0/4247151447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7741544b0 msgr2=0x7ff774155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:27.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.808+0000 7ff773fff640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7741544b0 0x7ff774155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:27.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.809+0000 7ff773fff640 1 -- 172.21.15.129:0/4247151447 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff760009400 con 0x7ff7740caa00 2024-03-23T15:47:27.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.811+0000 7ff773fff640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff774154bd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff7600098c0 tx=0x7ff760004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:27.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.815+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76003e040 con 0x7ff7740caa00 2024-03-23T15:47:27.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.815+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff760039cd0 con 0x7ff7740caa00 2024-03-23T15:47:27.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.815+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760044630 con 0x7ff7740caa00 2024-03-23T15:47:27.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.815+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff774159870 con 0x7ff7740caa00 2024-03-23T15:47:27.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.815+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7741576e0 con 0x7ff7740caa00 2024-03-23T15:47:27.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.816+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff774075740 con 0x7ff7740caa00 2024-03-23T15:47:27.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.834+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff76004d3b0 con 0x7ff7740caa00 2024-03-23T15:47:27.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.835+0000 7ff7717fa640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff75806b810 0x7ff75806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:27.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.835+0000 7ff778882640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff75806b810 0x7ff75806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:27.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.836+0000 7ff778882640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff75806b810 0x7ff75806dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff7680098e0 tx=0x7ff768009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:27.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:27.839+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff760039e40 con 0x7ff7740caa00 2024-03-23T15:47:28.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.075+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff7600775d0 con 0x7ff7740caa00 2024-03-23T15:47:28.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.255+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff774157d00 con 0x7ff7740caa00 2024-03-23T15:47:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.364+0000 7ff7717fa640 1 -- 172.21.15.129:0/4247151447 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16658 (secure 0 0 0) 0x7ff76007ac90 con 0x7ff7740caa00 2024-03-23T15:47:28.367 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:28.367 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:26.803695+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff75806b810 msgr2=0x7ff75806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff75806b810 0x7ff75806dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff7680098e0 tx=0x7ff768009060 comp rx=0 tx=0).stop 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 msgr2=0x7ff774154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff774154bd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff7600098c0 tx=0x7ff760004730 comp rx=0 tx=0).stop 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 shutdown_connections 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7741544b0 0x7ff774155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7740caa00 0x7ff774154bd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff75806b810 0x7ff75806dcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 --2- 172.21.15.129:0/4247151447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff77406c0b0 0x7ff7741560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 >> 172.21.15.129:0/4247151447 conn(0x7ff774061b50 msgr2=0x7ff7740c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 shutdown_connections 2024-03-23T15:47:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.366+0000 7ff77aae6640 1 -- 172.21.15.129:0/4247151447 wait complete. 2024-03-23T15:47:28.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.481+0000 7f82e76b5640 1 Processor -- start 2024-03-23T15:47:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.481+0000 7f82e76b5640 1 -- start start 2024-03-23T15:47:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.482+0000 7f82e76b5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f82e0071530 0x7f82e0158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.482+0000 7f82e76b5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e01591e0 con 0x7f82e0141630 2024-03-23T15:47:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.482+0000 7f82e76b5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e0141a10 con 0x7f82e00c8f70 2024-03-23T15:47:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.482+0000 7f82e76b5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f82e015c740 con 0x7f82e0071530 2024-03-23T15:47:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.483+0000 7f82e5c52640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f82e0071530 0x7f82e0158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.483+0000 7f82e4c50640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f82e00c8f70 0x7f82e00c4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55966/0 (socket says 172.21.15.129:55966) 2024-03-23T15:47:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.483+0000 7f82e4c50640 1 -- 172.21.15.129:0/3152566527 learned_addr learned my addr 172.21.15.129:0/3152566527 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.488+0000 7f82e5c52640 1 -- 172.21.15.129:0/3152566527 >> v1:172.21.15.157:6789/0 conn(0x7f82e00c8f70 legacy=0x7f82e00c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.488+0000 7f82e5c52640 1 -- 172.21.15.129:0/3152566527 >> v1:172.21.15.129:6789/0 conn(0x7f82e0141630 legacy=0x7f82e015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.489+0000 7f82e5c52640 1 -- 172.21.15.129:0/3152566527 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e015bba0 con 0x7f82e0071530 2024-03-23T15:47:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.493+0000 7f82e5c52640 1 --2- 172.21.15.129:0/3152566527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0071530 0x7f82e0158ca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82d00097a0 tx=0x7f82d002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403e257ddf275745 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:28.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.497+0000 7f82dffff640 1 -- 172.21.15.129:0/3152566527 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0031830 con 0x7f82e0071530 2024-03-23T15:47:28.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.500+0000 7f82dffff640 1 -- 172.21.15.129:0/3152566527 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82d0031990 con 0x7f82e0071530 2024-03-23T15:47:28.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.500+0000 7f82dffff640 1 -- 172.21.15.129:0/3152566527 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0031d20 con 0x7f82e0071530 2024-03-23T15:47:28.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 -- 172.21.15.129:0/3152566527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0071530 msgr2=0x7f82e0158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 --2- 172.21.15.129:0/3152566527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0071530 0x7f82e0158ca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82d00097a0 tx=0x7f82d002b8d0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 -- 172.21.15.129:0/3152566527 shutdown_connections 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 --2- 172.21.15.129:0/3152566527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0071530 0x7f82e0158ca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 -- 172.21.15.129:0/3152566527 >> 172.21.15.129:0/3152566527 conn(0x7f82e00c8370 msgr2=0x7f82e00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 -- 172.21.15.129:0/3152566527 shutdown_connections 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.501+0000 7f82e76b5640 1 -- 172.21.15.129:0/3152566527 wait complete. 2024-03-23T15:47:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.502+0000 7f82e76b5640 1 Processor -- start 2024-03-23T15:47:28.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.502+0000 7f82e76b5640 1 -- start start 2024-03-23T15:47:28.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e76b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82e0071530 0x7f82e01f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:28.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e76b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e00c8f70 0x7f82e0158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:28.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e76b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 0x7f82e00cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:28.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e76b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82e0157540 con 0x7f82e00c8f70 2024-03-23T15:47:28.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e76b5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f82e0159090 con 0x7f82e0141630 2024-03-23T15:47:28.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e76b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f82e01559d0 con 0x7f82e0071530 2024-03-23T15:47:28.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.503+0000 7f82e5c52640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 0x7f82e00cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:28.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.504+0000 7f82e5c52640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 0x7f82e00cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52690/0 (socket says 172.21.15.129:52690) 2024-03-23T15:47:28.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.504+0000 7f82e5c52640 1 -- 172.21.15.129:0/2331898087 learned_addr learned my addr 172.21.15.129:0/2331898087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:28.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.507+0000 7f82e5451640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82e0071530 0x7f82e01f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:28.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.508+0000 7f82e5c52640 1 -- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82e0071530 msgr2=0x7f82e01f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:28.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.508+0000 7f82e5c52640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82e0071530 0x7f82e01f1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.508+0000 7f82e5c52640 1 -- 172.21.15.129:0/2331898087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e00c8f70 msgr2=0x7f82e0158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:28.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.508+0000 7f82e5c52640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e00c8f70 0x7f82e0158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:28.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.508+0000 7f82e5c52640 1 -- 172.21.15.129:0/2331898087 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d0009400 con 0x7f82e0141630 2024-03-23T15:47:28.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.510+0000 7f82e5c52640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 0x7f82e00cabe0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f82d00097a0 tx=0x7f82d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:28.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.515+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0006e50 con 0x7f82e0141630 2024-03-23T15:47:28.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.515+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82d003aa60 con 0x7f82e0141630 2024-03-23T15:47:28.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.515+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e0155450 con 0x7f82e0141630 2024-03-23T15:47:28.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.515+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d003adf0 con 0x7f82e0141630 2024-03-23T15:47:28.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.515+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e0158ad0 con 0x7f82e0141630 2024-03-23T15:47:28.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.516+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e013ff40 con 0x7f82e0141630 2024-03-23T15:47:28.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.535+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f82d004d450 con 0x7f82e0141630 2024-03-23T15:47:28.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.535+0000 7f82ddffb640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f82cc06b810 0x7f82cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:28.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.536+0000 7f82e5451640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f82cc06b810 0x7f82cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:28.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.536+0000 7f82e5451640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f82cc06b810 0x7f82cc06dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007fd0 tx=0x7f82d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:28.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.773+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f82d007af90 con 0x7f82e0141630 2024-03-23T15:47:28.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.784+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82d0077230 con 0x7f82e0141630 2024-03-23T15:47:28.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:28.977+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82e01591e0 con 0x7f82e0141630 2024-03-23T15:47:29.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.020+0000 7f82ddffb640 1 -- 172.21.15.129:0/2331898087 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f82e01591e0 con 0x7f82e0141630 2024-03-23T15:47:29.026 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:29.026 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:29.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f82cc06b810 msgr2=0x7f82cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f82cc06b810 0x7f82cc06dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007fd0 tx=0x7f82d8009040 comp rx=0 tx=0).stop 2024-03-23T15:47:29.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 msgr2=0x7f82e00cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 0x7f82e00cabe0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f82d00097a0 tx=0x7f82d00046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 shutdown_connections 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f82cc06b810 0x7f82cc06dcd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82e0141630 0x7f82e00cabe0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e00c8f70 0x7f82e0158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 --2- 172.21.15.129:0/2331898087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82e0071530 0x7f82e01f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 >> 172.21.15.129:0/2331898087 conn(0x7f82e00c8370 msgr2=0x7f82e0151dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 shutdown_connections 2024-03-23T15:47:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.022+0000 7f82e76b5640 1 -- 172.21.15.129:0/2331898087 wait complete. 2024-03-23T15:47:29.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T15:47:29.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.137+0000 7feee32eb640 1 Processor -- start 2024-03-23T15:47:29.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.137+0000 7feee32eb640 1 -- start start 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.137+0000 7feee32eb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7feedc0cc2f0 0x7feedc071a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.138+0000 7feee32eb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc0c2d00 con 0x7feedc0715c0 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.138+0000 7feee32eb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc0c35a0 con 0x7feedc140390 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.138+0000 7feee32eb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7feedc0c3a30 con 0x7feedc0cc2f0 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.138+0000 7feee1888640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7feedc0cc2f0 0x7feedc071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.138+0000 7feee1087640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7feedc140390 0x7feedc0c5070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55980/0 (socket says 172.21.15.129:55980) 2024-03-23T15:47:29.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.138+0000 7feee1087640 1 -- 172.21.15.129:0/4008947936 learned_addr learned my addr 172.21.15.129:0/4008947936 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:29.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.141+0000 7feed3fff640 1 -- 172.21.15.129:0/4008947936 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091196835 0 0) 0x7feedc0c35a0 con 0x7feedc140390 2024-03-23T15:47:29.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.141+0000 7feed3fff640 1 -- 172.21.15.129:0/4008947936 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feec8003520 con 0x7feedc140390 2024-03-23T15:47:29.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.143+0000 7feee1888640 1 -- 172.21.15.129:0/4008947936 >> v1:172.21.15.157:6789/0 conn(0x7feedc140390 legacy=0x7feedc0c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.143+0000 7feee1888640 1 -- 172.21.15.129:0/4008947936 >> v1:172.21.15.129:6789/0 conn(0x7feedc0715c0 legacy=0x7feedc15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.143+0000 7feee1888640 1 -- 172.21.15.129:0/4008947936 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc0c58b0 con 0x7feedc0cc2f0 2024-03-23T15:47:29.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.146+0000 7feee1888640 1 --2- 172.21.15.129:0/4008947936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc071a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feed4009d90 tx=0x7feed402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3134f89e54c04e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:29.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.148+0000 7feed3fff640 1 -- 172.21.15.129:0/4008947936 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed402cd00 con 0x7feedc0cc2f0 2024-03-23T15:47:29.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feed3fff640 1 -- 172.21.15.129:0/4008947936 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed402ce60 con 0x7feedc0cc2f0 2024-03-23T15:47:29.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feed3fff640 1 -- 172.21.15.129:0/4008947936 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4034640 con 0x7feedc0cc2f0 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 -- 172.21.15.129:0/4008947936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 msgr2=0x7feedc071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 --2- 172.21.15.129:0/4008947936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc071a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feed4009d90 tx=0x7feed402bf50 comp rx=0 tx=0).stop 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 -- 172.21.15.129:0/4008947936 shutdown_connections 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 --2- 172.21.15.129:0/4008947936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc071a40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 -- 172.21.15.129:0/4008947936 >> 172.21.15.129:0/4008947936 conn(0x7feedc0cb800 msgr2=0x7feedc0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 -- 172.21.15.129:0/4008947936 shutdown_connections 2024-03-23T15:47:29.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.150+0000 7feee32eb640 1 -- 172.21.15.129:0/4008947936 wait complete. 2024-03-23T15:47:29.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 Processor -- start 2024-03-23T15:47:29.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 -- start start 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc0715c0 0x7feedc1ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc1383e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feedc140390 0x7feedc15a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feedc13c000 con 0x7feedc0cc2f0 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feedc1ed930 con 0x7feedc0715c0 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.152+0000 7feee32eb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feedc13bb70 con 0x7feedc140390 2024-03-23T15:47:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.153+0000 7feee0886640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc1383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:29.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.153+0000 7feee1087640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc0715c0 0x7feedc1ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:29.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.153+0000 7feee1888640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feedc140390 0x7feedc15a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:29.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.153+0000 7feee0886640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc1383e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37534/0 (socket says 172.21.15.129:37534) 2024-03-23T15:47:29.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.153+0000 7feee0886640 1 -- 172.21.15.129:0/1192188845 learned_addr learned my addr 172.21.15.129:0/1192188845 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:29.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.156+0000 7feee0886640 1 -- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feedc140390 msgr2=0x7feedc15a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.156+0000 7feee0886640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feedc140390 0x7feedc15a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.156+0000 7feee0886640 1 -- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc0715c0 msgr2=0x7feedc1ed3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:29.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.156+0000 7feee0886640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc0715c0 0x7feedc1ed3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:29.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.156+0000 7feee0886640 1 -- 172.21.15.129:0/1192188845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed4009a70 con 0x7feedc0cc2f0 2024-03-23T15:47:29.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.159+0000 7feee0886640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc1383e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c3c0 tx=0x7feecc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:29.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.162+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc0052d0 con 0x7feedc0cc2f0 2024-03-23T15:47:29.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.162+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feecc005450 con 0x7feedc0cc2f0 2024-03-23T15:47:29.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.162+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feecc007260 con 0x7feedc0cc2f0 2024-03-23T15:47:29.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.162+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feedc15a9d0 con 0x7feedc0cc2f0 2024-03-23T15:47:29.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.163+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feedc13b240 con 0x7feedc0cc2f0 2024-03-23T15:47:29.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.163+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feedc075740 con 0x7feedc0cc2f0 2024-03-23T15:47:29.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.181+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feecc0073c0 con 0x7feedc0cc2f0 2024-03-23T15:47:29.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.182+0000 7feed1ffb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec006daa0 0x7feec006ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:29.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.182+0000 7feee1087640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec006daa0 0x7feec006ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:29.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.183+0000 7feee1087640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec006daa0 0x7feec006ff60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feed80011e0 tx=0x7feed8007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:29.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.184+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7feecc058570 con 0x7feedc0cc2f0 2024-03-23T15:47:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.418+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feecc054810 con 0x7feedc0cc2f0 2024-03-23T15:47:29.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:29.647+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7feedc13c1e0 con 0x7feedc0cc2f0 2024-03-23T15:47:30.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.813+0000 7feed1ffb640 1 -- 172.21.15.129:0/1192188845 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7feecc020020 con 0x7feedc0cc2f0 2024-03-23T15:47:30.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec006daa0 msgr2=0x7feec006ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:30.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec006daa0 0x7feec006ff60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feed80011e0 tx=0x7feed8007d30 comp rx=0 tx=0).stop 2024-03-23T15:47:30.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 msgr2=0x7feedc1383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:30.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc1383e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c3c0 tx=0x7feecc00c880 comp rx=0 tx=0).stop 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 shutdown_connections 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feedc140390 0x7feedc15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feedc0cc2f0 0x7feedc1383e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec006daa0 0x7feec006ff60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 --2- 172.21.15.129:0/1192188845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feedc0715c0 0x7feedc1ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 >> 172.21.15.129:0/1192188845 conn(0x7feedc0cb800 msgr2=0x7feedc072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 shutdown_connections 2024-03-23T15:47:30.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.815+0000 7feee32eb640 1 -- 172.21.15.129:0/1192188845 wait complete. 2024-03-23T15:47:30.829 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-03-23T15:47:30.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-23T15:47:30.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.933+0000 7f783ceae640 1 Processor -- start 2024-03-23T15:47:30.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.933+0000 7f783ceae640 1 -- start start 2024-03-23T15:47:30.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.934+0000 7f783ceae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f78380ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:30.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.934+0000 7f783ceae640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f783815be00 0x7f783815c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:30.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.934+0000 7f783ceae640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f783815b970 con 0x7f783815b110 2024-03-23T15:47:30.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.934+0000 7f783ceae640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f783815c7d0 con 0x7f783815be00 2024-03-23T15:47:30.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.934+0000 7f783ceae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f783815d6d0 con 0x7f783815d2f0 2024-03-23T15:47:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.962+0000 7f7836d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f783815be00 0x7f783815c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.962+0000 7f7835d9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f783815b110 0x7f783806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33198/0 (socket says 172.21.15.129:33198) 2024-03-23T15:47:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.962+0000 7f7835d9b640 1 -- 172.21.15.129:0/3978575349 learned_addr learned my addr 172.21.15.129:0/3978575349 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.965+0000 7f7836d9d640 1 -- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 msgr2=0x7f78380ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.965+0000 7f7836d9d640 1 --2- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f78380ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.965+0000 7f7836d9d640 1 -- 172.21.15.129:0/3978575349 >> v1:172.21.15.157:6789/0 conn(0x7f783815b110 legacy=0x7f783806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.965+0000 7f7836d9d640 1 -- 172.21.15.129:0/3978575349 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78380cdcd0 con 0x7f783815be00 2024-03-23T15:47:30.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.971+0000 7f7836d9d640 1 --2- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815be00 0x7f783815c290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78280097a0 tx=0x7f782802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37392f6739ca665d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:30.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.975+0000 7f783559a640 1 -- 172.21.15.129:0/3978575349 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828030610 con 0x7f783815be00 2024-03-23T15:47:30.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.975+0000 7f783559a640 1 -- 172.21.15.129:0/3978575349 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7828030770 con 0x7f783815be00 2024-03-23T15:47:30.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.975+0000 7f783559a640 1 -- 172.21.15.129:0/3978575349 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828030b00 con 0x7f783815be00 2024-03-23T15:47:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 -- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815be00 msgr2=0x7f783815c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 --2- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815be00 0x7f783815c290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78280097a0 tx=0x7f782802b960 comp rx=0 tx=0).stop 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 -- 172.21.15.129:0/3978575349 shutdown_connections 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 --2- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815be00 0x7f783815c290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 --2- 172.21.15.129:0/3978575349 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f78380ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 -- 172.21.15.129:0/3978575349 >> 172.21.15.129:0/3978575349 conn(0x7f78380b1840 msgr2=0x7f78380b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 -- 172.21.15.129:0/3978575349 shutdown_connections 2024-03-23T15:47:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.976+0000 7f783ceae640 1 -- 172.21.15.129:0/3978575349 wait complete. 2024-03-23T15:47:30.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.978+0000 7f783ceae640 1 Processor -- start 2024-03-23T15:47:30.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.978+0000 7f783ceae640 1 -- start start 2024-03-23T15:47:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.979+0000 7f783ceae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815b110 0x7f78380c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.979+0000 7f783ceae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f783815be00 0x7f78380cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.979+0000 7f783ceae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f783814f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.979+0000 7f783ceae640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78380c3bf0 con 0x7f783815be00 2024-03-23T15:47:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.979+0000 7f783ceae640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f78381574f0 con 0x7f783815b110 2024-03-23T15:47:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.979+0000 7f783ceae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f783814ebb0 con 0x7f783815d2f0 2024-03-23T15:47:30.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.980+0000 7f783659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815b110 0x7f78380c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:30.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.980+0000 7f7836d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f783814f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:30.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.980+0000 7f7835d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f783815be00 0x7f78380cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:30.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.980+0000 7f7836d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f783814f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35482/0 (socket says 172.21.15.129:35482) 2024-03-23T15:47:30.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.980+0000 7f7836d9d640 1 -- 172.21.15.129:0/559954473 learned_addr learned my addr 172.21.15.129:0/559954473 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:30.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.984+0000 7f7835d9b640 1 -- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 msgr2=0x7f783814f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:30.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.984+0000 7f7835d9b640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f783814f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.984+0000 7f7835d9b640 1 -- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815b110 msgr2=0x7f78380c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.984+0000 7f7835d9b640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815b110 0x7f78380c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.984+0000 7f7835d9b640 1 -- 172.21.15.129:0/559954473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7828009400 con 0x7f783815be00 2024-03-23T15:47:30.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.986+0000 7f7835d9b640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f783815be00 0x7f78380cc790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f782c00ca50 tx=0x7f782c00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:30.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.989+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c00abb0 con 0x7f783815be00 2024-03-23T15:47:30.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.989+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783814ee90 con 0x7f783815be00 2024-03-23T15:47:30.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.989+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7838157740 con 0x7f783815be00 2024-03-23T15:47:30.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.990+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783804f530 con 0x7f783815be00 2024-03-23T15:47:30.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.990+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782c001b40 con 0x7f783815be00 2024-03-23T15:47:30.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:30.990+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c004b60 con 0x7f783815be00 2024-03-23T15:47:31.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.258+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f782c012400 con 0x7f783815be00 2024-03-23T15:47:31.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.258+0000 7f7826ffd640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781806b810 0x7f781806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:31.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.259+0000 7f783659c640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781806b810 0x7f781806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:31.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.259+0000 7f783659c640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781806b810 0x7f781806dcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78300098e0 tx=0x7f7830009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.272+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f782c058cd0 con 0x7f783815be00 2024-03-23T15:47:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.272+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f782c054f70 con 0x7f783815be00 2024-03-23T15:47:31.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.491+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f783814f020 con 0x7f783815be00 2024-03-23T15:47:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.992+0000 7f7826ffd640 1 -- 172.21.15.129:0/559954473 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v8) ==== 105+0+0 (secure 0 0 0) 0x7f782c058630 con 0x7f783815be00 2024-03-23T15:47:31.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781806b810 msgr2=0x7f781806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:31.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781806b810 0x7f781806dcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78300098e0 tx=0x7f7830009060 comp rx=0 tx=0).stop 2024-03-23T15:47:31.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f783815be00 msgr2=0x7f78380cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:31.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f783815be00 0x7f78380cc790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f782c00ca50 tx=0x7f782c00cf10 comp rx=0 tx=0).stop 2024-03-23T15:47:31.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 shutdown_connections 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815d2f0 0x7f783814f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f783815be00 0x7f78380cc790 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781806b810 0x7f781806dcd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 --2- 172.21.15.129:0/559954473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815b110 0x7f78380c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 >> 172.21.15.129:0/559954473 conn(0x7f78380b1840 msgr2=0x7f78380b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 shutdown_connections 2024-03-23T15:47:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:31.994+0000 7f783ceae640 1 -- 172.21.15.129:0/559954473 wait complete. 2024-03-23T15:47:32.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T15:47:32.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.111+0000 7fb9ed08f640 1 Processor -- start 2024-03-23T15:47:32.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.111+0000 7fb9ed08f640 1 -- start start 2024-03-23T15:47:32.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.112+0000 7fb9ed08f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb9e815d4f0 0x7fb9e806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.112+0000 7fb9ed08f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e80cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.112+0000 7fb9ed08f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e8154c90 con 0x7fb9e815b940 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.112+0000 7fb9ed08f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb9e806cb20 con 0x7fb9e815d4f0 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.112+0000 7fb9ed08f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e806bb90 con 0x7fb9e806d570 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.113+0000 7fb9e759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e80cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.113+0000 7fb9e759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e80cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35490/0 (socket says 172.21.15.129:35490) 2024-03-23T15:47:32.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.113+0000 7fb9e759e640 1 -- 172.21.15.129:0/3549793679 learned_addr learned my addr 172.21.15.129:0/3549793679 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:32.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.115+0000 7fb9e759e640 1 -- 172.21.15.129:0/3549793679 >> v2:172.21.15.157:3300/0 conn(0x7fb9e815d4f0 msgr2=0x7fb9e806c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:32.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.115+0000 7fb9e759e640 1 --2- 172.21.15.129:0/3549793679 >> v2:172.21.15.157:3300/0 conn(0x7fb9e815d4f0 0x7fb9e806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:32.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.115+0000 7fb9e759e640 1 -- 172.21.15.129:0/3549793679 >> v1:172.21.15.157:6789/0 conn(0x7fb9e815b940 legacy=0x7fb9e815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:32.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.115+0000 7fb9e759e640 1 -- 172.21.15.129:0/3549793679 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e806c150 con 0x7fb9e806d570 2024-03-23T15:47:32.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.117+0000 7fb9e759e640 1 --2- 172.21.15.129:0/3549793679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e80cc290 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009e50 tx=0x7fb9e002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b16c5508b8d6b123 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:32.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.121+0000 7fb9e5d9b640 1 -- 172.21.15.129:0/3549793679 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e002cb00 con 0x7fb9e806d570 2024-03-23T15:47:32.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.121+0000 7fb9e5d9b640 1 -- 172.21.15.129:0/3549793679 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9e002cc60 con 0x7fb9e806d570 2024-03-23T15:47:32.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.121+0000 7fb9e5d9b640 1 -- 172.21.15.129:0/3549793679 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e00343f0 con 0x7fb9e806d570 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 -- 172.21.15.129:0/3549793679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 msgr2=0x7fb9e80cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/3549793679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e80cc290 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0009e50 tx=0x7fb9e002bf60 comp rx=0 tx=0).stop 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 -- 172.21.15.129:0/3549793679 shutdown_connections 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/3549793679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e80cc290 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/3549793679 >> v2:172.21.15.157:3300/0 conn(0x7fb9e815d4f0 0x7fb9e806c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 -- 172.21.15.129:0/3549793679 >> 172.21.15.129:0/3549793679 conn(0x7fb9e80cfac0 msgr2=0x7fb9e80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 -- 172.21.15.129:0/3549793679 shutdown_connections 2024-03-23T15:47:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.122+0000 7fb9ed08f640 1 -- 172.21.15.129:0/3549793679 wait complete. 2024-03-23T15:47:32.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.123+0000 7fb9ed08f640 1 Processor -- start 2024-03-23T15:47:32.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.123+0000 7fb9ed08f640 1 -- start start 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9ed08f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e81f0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9ed08f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e815b940 0x7fb9e80b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9ed08f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9e815d4f0 0x7fb9e814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9ed08f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8151610 con 0x7fb9e815b940 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9ed08f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8151880 con 0x7fb9e815d4f0 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9ed08f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e8151b60 con 0x7fb9e806d570 2024-03-23T15:47:32.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.124+0000 7fb9e6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e81f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:32.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.125+0000 7fb9e6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e81f0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35502/0 (socket says 172.21.15.129:35502) 2024-03-23T15:47:32.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.125+0000 7fb9e6d9d640 1 -- 172.21.15.129:0/1316282217 learned_addr learned my addr 172.21.15.129:0/1316282217 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:32.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.128+0000 7fb9e6d9d640 1 -- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9e815d4f0 msgr2=0x7fb9e814f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:32.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.128+0000 7fb9e6d9d640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9e815d4f0 0x7fb9e814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:32.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.128+0000 7fb9e6d9d640 1 -- 172.21.15.129:0/1316282217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e815b940 msgr2=0x7fb9e80b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:32.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.128+0000 7fb9e6d9d640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e815b940 0x7fb9e80b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:32.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.128+0000 7fb9e6d9d640 1 -- 172.21.15.129:0/1316282217 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e0009b30 con 0x7fb9e806d570 2024-03-23T15:47:32.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.131+0000 7fb9e6d9d640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e81f0980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b410 tx=0x7fb9d800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.134+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8011830 con 0x7fb9e806d570 2024-03-23T15:47:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.134+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d8011990 con 0x7fb9e806d570 2024-03-23T15:47:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.134+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e814ee50 con 0x7fb9e806d570 2024-03-23T15:47:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.134+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8011d40 con 0x7fb9e806d570 2024-03-23T15:47:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.134+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e814f410 con 0x7fb9e806d570 2024-03-23T15:47:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.136+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e80686f0 con 0x7fb9e806d570 2024-03-23T15:47:32.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.385+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb9d801a400 con 0x7fb9e806d570 2024-03-23T15:47:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.385+0000 7fb9cf7fe640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb9d406b860 0x7fb9d406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.385+0000 7fb9e659c640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb9d406b860 0x7fb9d406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.386+0000 7fb9e659c640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb9d406b860 0x7fb9d406dd20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80c71c0 tx=0x7fb9dc007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:32.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.389+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb9d8059790 con 0x7fb9e806d570 2024-03-23T15:47:32.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.399+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb9d8055330 con 0x7fb9e806d570 2024-03-23T15:47:32.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:32.621+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb9e8075740 con 0x7fb9e806d570 2024-03-23T15:47:33.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.317+0000 7fb9cf7fe640 1 -- 172.21.15.129:0/1316282217 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fb9d8015020 con 0x7fb9e806d570 2024-03-23T15:47:33.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb9d406b860 msgr2=0x7fb9d406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb9d406b860 0x7fb9d406dd20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80c71c0 tx=0x7fb9dc007990 comp rx=0 tx=0).stop 2024-03-23T15:47:33.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 msgr2=0x7fb9e81f0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e81f0980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b410 tx=0x7fb9d800b8d0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 shutdown_connections 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb9d406b860 0x7fb9d406dd20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9e815d4f0 0x7fb9e814f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e815b940 0x7fb9e80b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 --2- 172.21.15.129:0/1316282217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9e806d570 0x7fb9e81f0980 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 >> 172.21.15.129:0/1316282217 conn(0x7fb9e80cfac0 msgr2=0x7fb9e80c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 shutdown_connections 2024-03-23T15:47:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.319+0000 7fb9ed08f640 1 -- 172.21.15.129:0/1316282217 wait complete. 2024-03-23T15:47:33.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-23T15:47:33.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.436+0000 7f9facf8a640 1 Processor -- start 2024-03-23T15:47:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.436+0000 7f9facf8a640 1 -- start start 2024-03-23T15:47:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9facf8a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9fa81437d0 0x7f9fa80c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9facf8a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa80c8770 con 0x7f9fa815d780 2024-03-23T15:47:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9facf8a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa806c240 con 0x7f9fa8070330 2024-03-23T15:47:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9facf8a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa815e7a0 con 0x7f9fa81437d0 2024-03-23T15:47:33.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9fa6d9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9fa815d780 0x7f9fa8164b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36502/0 (socket says 172.21.15.129:36502) 2024-03-23T15:47:33.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9fa659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9fa81437d0 0x7f9fa80c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:33.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.437+0000 7f9fa6d9d640 1 -- 172.21.15.129:0/4241526220 learned_addr learned my addr 172.21.15.129:0/4241526220 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:33.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.440+0000 7f9fa559a640 1 -- 172.21.15.129:0/4241526220 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639904266 0 0) 0x7f9fa0001490 con 0x7f9fa815d780 2024-03-23T15:47:33.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.440+0000 7f9fa559a640 1 -- 172.21.15.129:0/4241526220 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f90003520 con 0x7f9fa815d780 2024-03-23T15:47:33.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.441+0000 7f9fa659c640 1 -- 172.21.15.129:0/4241526220 >> v1:172.21.15.157:6789/0 conn(0x7f9fa8070330 legacy=0x7f9fa80c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.441+0000 7f9fa659c640 1 -- 172.21.15.129:0/4241526220 >> v1:172.21.15.129:6789/0 conn(0x7f9fa815d780 legacy=0x7f9fa8164b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.441+0000 7f9fa659c640 1 -- 172.21.15.129:0/4241526220 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa806fc70 con 0x7f9fa81437d0 2024-03-23T15:47:33.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.443+0000 7f9fa659c640 1 --2- 172.21.15.129:0/4241526220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa81437d0 0x7f9fa80c8230 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f980097a0 tx=0x7f9f9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39444cf08153a5a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.448+0000 7f9fa559a640 1 -- 172.21.15.129:0/4241526220 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98031830 con 0x7f9fa81437d0 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.448+0000 7f9fa559a640 1 -- 172.21.15.129:0/4241526220 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f98031990 con 0x7f9fa81437d0 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.448+0000 7f9fa559a640 1 -- 172.21.15.129:0/4241526220 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98031d20 con 0x7f9fa81437d0 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 -- 172.21.15.129:0/4241526220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa81437d0 msgr2=0x7f9fa80c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 --2- 172.21.15.129:0/4241526220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa81437d0 0x7f9fa80c8230 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f980097a0 tx=0x7f9f9802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 -- 172.21.15.129:0/4241526220 shutdown_connections 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 --2- 172.21.15.129:0/4241526220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa81437d0 0x7f9fa80c8230 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 -- 172.21.15.129:0/4241526220 >> 172.21.15.129:0/4241526220 conn(0x7f9fa80718f0 msgr2=0x7f9fa8061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 -- 172.21.15.129:0/4241526220 shutdown_connections 2024-03-23T15:47:33.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.449+0000 7f9facf8a640 1 -- 172.21.15.129:0/4241526220 wait complete. 2024-03-23T15:47:33.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.450+0000 7f9facf8a640 1 Processor -- start 2024-03-23T15:47:33.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.450+0000 7f9facf8a640 1 -- start start 2024-03-23T15:47:33.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.451+0000 7f9facf8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa8070330 0x7f9fa81f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:33.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.451+0000 7f9facf8a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9fa81437d0 0x7f9fa81e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:33.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.451+0000 7f9facf8a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fa815d780 0x7f9fa813bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:33.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.451+0000 7f9facf8a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa813df10 con 0x7f9fa815d780 2024-03-23T15:47:33.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.451+0000 7f9facf8a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa813db00 con 0x7f9fa8070330 2024-03-23T15:47:33.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.451+0000 7f9facf8a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa81f0380 con 0x7f9fa81437d0 2024-03-23T15:47:33.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.452+0000 7f9fa6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fa815d780 0x7f9fa813bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:33.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.452+0000 7f9fa5d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9fa81437d0 0x7f9fa81e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:33.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.452+0000 7f9fa659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa8070330 0x7f9fa81f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:33.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.452+0000 7f9fa5d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9fa81437d0 0x7f9fa81e3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35514/0 (socket says 172.21.15.129:35514) 2024-03-23T15:47:33.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.452+0000 7f9fa5d9b640 1 -- 172.21.15.129:0/3806509552 learned_addr learned my addr 172.21.15.129:0/3806509552 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:33.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.455+0000 7f9fa6d9d640 1 -- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9fa81437d0 msgr2=0x7f9fa81e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.455+0000 7f9fa6d9d640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9fa81437d0 0x7f9fa81e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.455+0000 7f9fa6d9d640 1 -- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa8070330 msgr2=0x7f9fa81f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:33.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.455+0000 7f9fa6d9d640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa8070330 0x7f9fa81f0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:33.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.455+0000 7f9fa6d9d640 1 -- 172.21.15.129:0/3806509552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98009400 con 0x7f9fa815d780 2024-03-23T15:47:33.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.458+0000 7f9fa6d9d640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fa815d780 0x7f9fa813bbe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000caa0 tx=0x7f9fa000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:33.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.461+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa000ac00 con 0x7f9fa815d780 2024-03-23T15:47:33.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.461+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa000ad60 con 0x7f9fa815d780 2024-03-23T15:47:33.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.461+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa813d260 con 0x7f9fa815d780 2024-03-23T15:47:33.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.461+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0012620 con 0x7f9fa815d780 2024-03-23T15:47:33.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.461+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa81f0640 con 0x7f9fa815d780 2024-03-23T15:47:33.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.462+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa81f0860 con 0x7f9fa815d780 2024-03-23T15:47:33.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.749+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9fa0007070 con 0x7f9fa815d780 2024-03-23T15:47:33.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.749+0000 7f9f96ffd640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9f8806b890 0x7f9f8806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:33.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.750+0000 7f9fa659c640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9f8806b890 0x7f9f8806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:33.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.750+0000 7f9fa659c640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9f8806b890 0x7f9f8806dd50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f98036000 tx=0x7f9f9803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.765+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9fa0058b80 con 0x7f9fa815d780 2024-03-23T15:47:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.765+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9fa0054e20 con 0x7f9fa815d780 2024-03-23T15:47:33.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:33.989+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f9fa80c8770 con 0x7f9fa815d780 2024-03-23T15:47:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.272+0000 7f9f96ffd640 1 -- 172.21.15.129:0/3806509552 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v11) ==== 115+0+0 (secure 0 0 0) 0x7f9fa80c8770 con 0x7f9fa815d780 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9f8806b890 msgr2=0x7f9f8806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9f8806b890 0x7f9f8806dd50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f98036000 tx=0x7f9f9803c040 comp rx=0 tx=0).stop 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fa815d780 msgr2=0x7f9fa813bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fa815d780 0x7f9fa813bbe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000caa0 tx=0x7f9fa000cf60 comp rx=0 tx=0).stop 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 shutdown_connections 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fa815d780 0x7f9fa813bbe0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9fa81437d0 0x7f9fa81e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9f8806b890 0x7f9f8806dd50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 --2- 172.21.15.129:0/3806509552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fa8070330 0x7f9fa81f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 >> 172.21.15.129:0/3806509552 conn(0x7f9fa80718f0 msgr2=0x7f9fa80cbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:34.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 shutdown_connections 2024-03-23T15:47:34.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.274+0000 7f9facf8a640 1 -- 172.21.15.129:0/3806509552 wait complete. 2024-03-23T15:47:34.288 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T15:47:34.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:34.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.390+0000 7f40c4f56640 1 Processor -- start 2024-03-23T15:47:34.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.390+0000 7f40c4f56640 1 -- start start 2024-03-23T15:47:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.391+0000 7f40c4f56640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 0x7f40c00cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.391+0000 7f40c4f56640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f40c015df80 0x7f40c01540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.391+0000 7f40c4f56640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c0070640 con 0x7f40c015dba0 2024-03-23T15:47:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.391+0000 7f40c4f56640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f40c00c4c00 con 0x7f40c015df80 2024-03-23T15:47:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.391+0000 7f40c4f56640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40c006c240 con 0x7f40c006cf70 2024-03-23T15:47:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.392+0000 7f40bed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f40c015df80 0x7f40c01540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.392+0000 7f40bdd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 0x7f40c00cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.392+0000 7f40bdd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 0x7f40c00cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35526/0 (socket says 172.21.15.129:35526) 2024-03-23T15:47:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.392+0000 7f40bdd9b640 1 -- 172.21.15.129:0/1183670926 learned_addr learned my addr 172.21.15.129:0/1183670926 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.395+0000 7f40bdd9b640 1 -- 172.21.15.129:0/1183670926 >> v2:172.21.15.157:3300/0 conn(0x7f40c015df80 msgr2=0x7f40c01540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.395+0000 7f40bdd9b640 1 --2- 172.21.15.129:0/1183670926 >> v2:172.21.15.157:3300/0 conn(0x7f40c015df80 0x7f40c01540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.395+0000 7f40bdd9b640 1 -- 172.21.15.129:0/1183670926 >> v1:172.21.15.157:6789/0 conn(0x7f40c015dba0 legacy=0x7f40c006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.395+0000 7f40bdd9b640 1 -- 172.21.15.129:0/1183670926 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c00cbb20 con 0x7f40c006cf70 2024-03-23T15:47:34.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.397+0000 7f40bdd9b640 1 --2- 172.21.15.129:0/1183670926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 0x7f40c00cca30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009850 tx=0x7f40b402b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6be423a6ee327a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.401+0000 7f40bd59a640 1 -- 172.21.15.129:0/1183670926 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004230 con 0x7f40c006cf70 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.401+0000 7f40bd59a640 1 -- 172.21.15.129:0/1183670926 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b4004390 con 0x7f40c006cf70 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.401+0000 7f40bd59a640 1 -- 172.21.15.129:0/1183670926 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4003810 con 0x7f40c006cf70 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 -- 172.21.15.129:0/1183670926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 msgr2=0x7f40c00cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1183670926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 0x7f40c00cca30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009850 tx=0x7f40b402b9f0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 -- 172.21.15.129:0/1183670926 shutdown_connections 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1183670926 >> v2:172.21.15.157:3300/0 conn(0x7f40c015df80 0x7f40c01540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1183670926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c006cf70 0x7f40c00cca30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 -- 172.21.15.129:0/1183670926 >> 172.21.15.129:0/1183670926 conn(0x7f40c0071300 msgr2=0x7f40c00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 -- 172.21.15.129:0/1183670926 shutdown_connections 2024-03-23T15:47:34.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.402+0000 7f40c4f56640 1 -- 172.21.15.129:0/1183670926 wait complete. 2024-03-23T15:47:34.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 Processor -- start 2024-03-23T15:47:34.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 -- start start 2024-03-23T15:47:34.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40c006cf70 0x7f40c01e5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:34.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40c015dba0 0x7f40c015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 0x7f40c00c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f40c01e5900 con 0x7f40c015dba0 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40c00c4e10 con 0x7f40c006cf70 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.404+0000 7f40c4f56640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40c00c7810 con 0x7f40c015df80 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.405+0000 7f40bed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 0x7f40c00c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.405+0000 7f40be59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40c006cf70 0x7f40c01e5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.405+0000 7f40bed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 0x7f40c00c6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35542/0 (socket says 172.21.15.129:35542) 2024-03-23T15:47:34.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.405+0000 7f40bed9d640 1 -- 172.21.15.129:0/1118309676 learned_addr learned my addr 172.21.15.129:0/1118309676 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:34.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.409+0000 7f40bed9d640 1 -- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40c006cf70 msgr2=0x7f40c01e5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.409+0000 7f40bed9d640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40c006cf70 0x7f40c01e5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.409+0000 7f40bed9d640 1 -- 172.21.15.129:0/1118309676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40c015dba0 msgr2=0x7f40c015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.409+0000 7f40bed9d640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40c015dba0 0x7f40c015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.409+0000 7f40bed9d640 1 -- 172.21.15.129:0/1118309676 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4009460 con 0x7f40c015df80 2024-03-23T15:47:34.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.412+0000 7f40bed9d640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 0x7f40c00c6480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f40b800efc0 tx=0x7f40b800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.416+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8009ba0 con 0x7f40c015df80 2024-03-23T15:47:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.416+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b800f040 con 0x7f40c015df80 2024-03-23T15:47:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.416+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c00c7db0 con 0x7f40c015df80 2024-03-23T15:47:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.416+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b80146c0 con 0x7f40c015df80 2024-03-23T15:47:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.416+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c00c7ad0 con 0x7f40c015df80 2024-03-23T15:47:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.417+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c00686f0 con 0x7f40c015df80 2024-03-23T15:47:34.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.435+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f40b8014820 con 0x7f40c015df80 2024-03-23T15:47:34.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.435+0000 7f40aeffd640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f40a006b860 0x7f40a006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:34.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.436+0000 7f40be59c640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f40a006b860 0x7f40a006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:34.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.436+0000 7f40be59c640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f40a006b860 0x7f40a006dd20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001780 tx=0x7f40b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:34.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.439+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f40b805a270 con 0x7f40c015df80 2024-03-23T15:47:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.690+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f40b8055e10 con 0x7f40c015df80 2024-03-23T15:47:34.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.952+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40c0075740 con 0x7f40c015df80 2024-03-23T15:47:34.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.975+0000 7f40aeffd640 1 -- 172.21.15.129:0/1118309676 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6877 (secure 0 0 0) 0x7f40b8018030 con 0x7f40c015df80 2024-03-23T15:47:34.978 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:34.979 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4234,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:47:34.266436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2],"up":{"mds_0":4253,"mds_1":4250,"mds_2":4246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:creating","state_seq":1,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6837/3877606926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3877606926},{"type":"v1","addr":"172.21.15.157:6837","nonce":3877606926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4253,4250]},"id":1}]} 2024-03-23T15:47:34.979 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 12 2024-03-23T15:47:34.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f40a006b860 msgr2=0x7f40a006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f40a006b860 0x7f40a006dd20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001780 tx=0x7f40b0009040 comp rx=0 tx=0).stop 2024-03-23T15:47:34.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 msgr2=0x7f40c00c6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 0x7f40c00c6480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f40b800efc0 tx=0x7f40b800c490 comp rx=0 tx=0).stop 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 shutdown_connections 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40c015df80 0x7f40c00c6480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40c015dba0 0x7f40c015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f40a006b860 0x7f40a006dd20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 --2- 172.21.15.129:0/1118309676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40c006cf70 0x7f40c01e5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.977+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 >> 172.21.15.129:0/1118309676 conn(0x7f40c0071300 msgr2=0x7f40c00caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.978+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 shutdown_connections 2024-03-23T15:47:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:34.978+0000 7f40c4f56640 1 -- 172.21.15.129:0/1118309676 wait complete. 2024-03-23T15:47:34.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:35.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 Processor -- start 2024-03-23T15:47:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 -- start start 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75601544b0 0x7f756015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f75600c6000 0x7f75600c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75600cbcb0 con 0x7f75600caa00 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f75600c5be0 con 0x7f75600c6000 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.091+0000 7f7564e55640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f75600c69b0 con 0x7f75601544b0 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.092+0000 7f755ed9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f75600c6000 0x7f75600c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.092+0000 7f755e59c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75601544b0 0x7f756015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.092+0000 7f755e59c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75601544b0 0x7f756015a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40992/0 (socket says 172.21.15.129:40992) 2024-03-23T15:47:35.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.092+0000 7f755e59c640 1 -- 172.21.15.129:0/2444064511 learned_addr learned my addr 172.21.15.129:0/2444064511 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:35.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.095+0000 7f755e59c640 1 -- 172.21.15.129:0/2444064511 >> v2:172.21.15.129:3300/0 conn(0x7f75600c6000 msgr2=0x7f75600c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:35.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.095+0000 7f755e59c640 1 --2- 172.21.15.129:0/2444064511 >> v2:172.21.15.129:3300/0 conn(0x7f75600c6000 0x7f75600c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.095+0000 7f755e59c640 1 -- 172.21.15.129:0/2444064511 >> v1:172.21.15.129:6789/0 conn(0x7f75600caa00 legacy=0x7f756006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.095+0000 7f755e59c640 1 -- 172.21.15.129:0/2444064511 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756015b280 con 0x7f75601544b0 2024-03-23T15:47:35.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.097+0000 7f755e59c640 1 --2- 172.21.15.129:0/2444064511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75601544b0 0x7f756015a2e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f756006cd70 tx=0x7f755002ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd04f7bf4e39fea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:35.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.100+0000 7f755d59a640 1 -- 172.21.15.129:0/2444064511 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755002f850 con 0x7f75601544b0 2024-03-23T15:47:35.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.100+0000 7f755d59a640 1 -- 172.21.15.129:0/2444064511 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f755002f9b0 con 0x7f75601544b0 2024-03-23T15:47:35.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.100+0000 7f755d59a640 1 -- 172.21.15.129:0/2444064511 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755002fd40 con 0x7f75601544b0 2024-03-23T15:47:35.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 -- 172.21.15.129:0/2444064511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75601544b0 msgr2=0x7f756015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 --2- 172.21.15.129:0/2444064511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75601544b0 0x7f756015a2e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f756006cd70 tx=0x7f755002ba30 comp rx=0 tx=0).stop 2024-03-23T15:47:35.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 -- 172.21.15.129:0/2444064511 shutdown_connections 2024-03-23T15:47:35.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 --2- 172.21.15.129:0/2444064511 >> v2:172.21.15.129:3300/0 conn(0x7f75600c6000 0x7f75600c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 --2- 172.21.15.129:0/2444064511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75601544b0 0x7f756015a2e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 -- 172.21.15.129:0/2444064511 >> 172.21.15.129:0/2444064511 conn(0x7f7560061b50 msgr2=0x7f7560061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:35.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 -- 172.21.15.129:0/2444064511 shutdown_connections 2024-03-23T15:47:35.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.101+0000 7f7564e55640 1 -- 172.21.15.129:0/2444064511 wait complete. 2024-03-23T15:47:35.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 Processor -- start 2024-03-23T15:47:35.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 -- start start 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 0x7f75601567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75600caa00 0x7f75601f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75601544b0 0x7f75601f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f75600c9b60 con 0x7f75600c6000 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7560157a40 con 0x7f75600caa00 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.103+0000 7f7564e55640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f75600c9830 con 0x7f75601544b0 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.104+0000 7f755dd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75600caa00 0x7f75601f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.104+0000 7f755e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 0x7f75601567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.104+0000 7f755ed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75601544b0 0x7f75601f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.104+0000 7f755e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 0x7f75601567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58792/0 (socket says 172.21.15.129:58792) 2024-03-23T15:47:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.104+0000 7f755e59c640 1 -- 172.21.15.129:0/1693056122 learned_addr learned my addr 172.21.15.129:0/1693056122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:35.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.108+0000 7f755e59c640 1 -- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75601544b0 msgr2=0x7f75601f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.108+0000 7f755e59c640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75601544b0 0x7f75601f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.108+0000 7f755e59c640 1 -- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75600caa00 msgr2=0x7f75601f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.108+0000 7f755e59c640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75600caa00 0x7f75601f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.108+0000 7f755e59c640 1 -- 172.21.15.129:0/1693056122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75500094d0 con 0x7f75600c6000 2024-03-23T15:47:35.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.110+0000 7f755e59c640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 0x7f75601567b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f755002ba70 tx=0x7f7550004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:35.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.115+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550037c40 con 0x7f75600c6000 2024-03-23T15:47:35.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.115+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7550037dc0 con 0x7f75600c6000 2024-03-23T15:47:35.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.115+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550041640 con 0x7f75600c6000 2024-03-23T15:47:35.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.115+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75601574a0 con 0x7f75600c6000 2024-03-23T15:47:35.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.115+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7560157760 con 0x7f75600c6000 2024-03-23T15:47:35.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.116+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75600b25e0 con 0x7f75600c6000 2024-03-23T15:47:35.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.140+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f75500417a0 con 0x7f75600c6000 2024-03-23T15:47:35.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.140+0000 7f754effd640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f754006b810 0x7f754006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.140+0000 7f755dd9b640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f754006b810 0x7f754006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.141+0000 7f755dd9b640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f754006b810 0x7f754006dcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f75600c8430 tx=0x7f7554009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:35.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.424+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f755003c070 con 0x7f75600c6000 2024-03-23T15:47:35.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.424+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7550078510 con 0x7f75600c6000 2024-03-23T15:47:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.649+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7560075740 con 0x7f75600c6000 2024-03-23T15:47:35.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.698+0000 7f754effd640 1 -- 172.21.15.129:0/1693056122 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f755007bb90 con 0x7f75600c6000 2024-03-23T15:47:35.701 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:35.702 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.700+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f754006b810 msgr2=0x7f754006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.700+0000 7f7564e55640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f754006b810 0x7f754006dcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f75600c8430 tx=0x7f7554009040 comp rx=0 tx=0).stop 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 msgr2=0x7f75601567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 0x7f75601567b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f755002ba70 tx=0x7f7550004730 comp rx=0 tx=0).stop 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 shutdown_connections 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75601544b0 0x7f75601f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f754006b810 0x7f754006dcd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75600caa00 0x7f75601f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 --2- 172.21.15.129:0/1693056122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75600c6000 0x7f75601567b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 >> 172.21.15.129:0/1693056122 conn(0x7f7560061b50 msgr2=0x7f7560158130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 shutdown_connections 2024-03-23T15:47:35.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.701+0000 7f7564e55640 1 -- 172.21.15.129:0/1693056122 wait complete. 2024-03-23T15:47:35.715 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T15:47:35.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T15:47:35.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:47:35.776 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:47:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.816+0000 7f36226c2640 1 Processor -- start 2024-03-23T15:47:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.817+0000 7f36226c2640 1 -- start start 2024-03-23T15:47:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.817+0000 7f36226c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.817+0000 7f36226c2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f361c0caa00 0x7f361c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.817+0000 7f36226c2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f361c0cbcb0 con 0x7f361c06c0b0 2024-03-23T15:47:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.817+0000 7f36226c2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f361c0c5be0 con 0x7f361c0caa00 2024-03-23T15:47:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.817+0000 7f36226c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f361c15b280 con 0x7f361c1544b0 2024-03-23T15:47:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.818+0000 7f361bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.818+0000 7f361b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f361c0caa00 0x7f361c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.818+0000 7f361bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c15a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35574/0 (socket says 172.21.15.129:35574) 2024-03-23T15:47:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.818+0000 7f361bfff640 1 -- 172.21.15.129:0/3901675847 learned_addr learned my addr 172.21.15.129:0/3901675847 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.820+0000 7f361bfff640 1 -- 172.21.15.129:0/3901675847 >> v2:172.21.15.157:3300/0 conn(0x7f361c0caa00 msgr2=0x7f361c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.821+0000 7f361bfff640 1 --2- 172.21.15.129:0/3901675847 >> v2:172.21.15.157:3300/0 conn(0x7f361c0caa00 0x7f361c06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.821+0000 7f361bfff640 1 -- 172.21.15.129:0/3901675847 >> v1:172.21.15.157:6789/0 conn(0x7f361c06c0b0 legacy=0x7f361c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.821+0000 7f361bfff640 1 -- 172.21.15.129:0/3901675847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c15a9e0 con 0x7f361c1544b0 2024-03-23T15:47:35.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.823+0000 7f361bfff640 1 --2- 172.21.15.129:0/3901675847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c15a2e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f360c009870 tx=0x7f360c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd12c4888234a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:35.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.827+0000 7f361affd640 1 -- 172.21.15.129:0/3901675847 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c02f840 con 0x7f361c1544b0 2024-03-23T15:47:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.876+0000 7f361affd640 1 -- 172.21.15.129:0/3901675847 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f360c02f9a0 con 0x7f361c1544b0 2024-03-23T15:47:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.876+0000 7f361affd640 1 -- 172.21.15.129:0/3901675847 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c02fd30 con 0x7f361c1544b0 2024-03-23T15:47:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 -- 172.21.15.129:0/3901675847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 msgr2=0x7f361c15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 --2- 172.21.15.129:0/3901675847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c15a2e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f360c009870 tx=0x7f360c02ba20 comp rx=0 tx=0).stop 2024-03-23T15:47:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 -- 172.21.15.129:0/3901675847 shutdown_connections 2024-03-23T15:47:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 --2- 172.21.15.129:0/3901675847 >> v2:172.21.15.157:3300/0 conn(0x7f361c0caa00 0x7f361c06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 --2- 172.21.15.129:0/3901675847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c15a2e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 -- 172.21.15.129:0/3901675847 >> 172.21.15.129:0/3901675847 conn(0x7f361c061b50 msgr2=0x7f361c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:35.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 -- 172.21.15.129:0/3901675847 shutdown_connections 2024-03-23T15:47:35.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.877+0000 7f36226c2640 1 -- 172.21.15.129:0/3901675847 wait complete. 2024-03-23T15:47:35.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.878+0000 7f36226c2640 1 Processor -- start 2024-03-23T15:47:35.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.878+0000 7f36226c2640 1 -- start start 2024-03-23T15:47:35.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f36226c2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f361c06c0b0 0x7f361c1efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f36226c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f361c0caa00 0x7f361c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f36226c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c1574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f36226c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f361c158130 con 0x7f361c0caa00 2024-03-23T15:47:35.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f36226c2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f361c154f30 con 0x7f361c06c0b0 2024-03-23T15:47:35.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f36226c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f361c155210 con 0x7f361c1544b0 2024-03-23T15:47:35.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.879+0000 7f3620c5f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c1574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.880+0000 7f361bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f361c06c0b0 0x7f361c1efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.880+0000 7f3620c5f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c1574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35590/0 (socket says 172.21.15.129:35590) 2024-03-23T15:47:35.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.880+0000 7f3620c5f640 1 -- 172.21.15.129:0/2590290923 learned_addr learned my addr 172.21.15.129:0/2590290923 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:35.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.882+0000 7f361b7fe640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f361c0caa00 0x7f361c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.883+0000 7f361bfff640 1 -- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 msgr2=0x7f361c1574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.883+0000 7f361bfff640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c1574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.883+0000 7f361bfff640 1 -- 172.21.15.129:0/2590290923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f361c0caa00 msgr2=0x7f361c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:35.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.883+0000 7f361bfff640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f361c0caa00 0x7f361c1566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:35.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.883+0000 7f361bfff640 1 -- 172.21.15.129:0/2590290923 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360c0094d0 con 0x7f361c06c0b0 2024-03-23T15:47:35.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.885+0000 7f361bfff640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f361c06c0b0 0x7f361c1efb20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f360c033000 tx=0x7f360c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:35.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.889+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c03c040 con 0x7f361c06c0b0 2024-03-23T15:47:35.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.889+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f360c037c60 con 0x7f361c06c0b0 2024-03-23T15:47:35.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.889+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361c1554f0 con 0x7f361c06c0b0 2024-03-23T15:47:35.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.889+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0423f0 con 0x7f361c06c0b0 2024-03-23T15:47:35.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.889+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361c155ab0 con 0x7f361c06c0b0 2024-03-23T15:47:35.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.890+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361c1582a0 con 0x7f361c06c0b0 2024-03-23T15:47:35.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.909+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f360c02d3d0 con 0x7f361c06c0b0 2024-03-23T15:47:35.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.910+0000 7f3618ff9640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f360806b840 0x7f360806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:35.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.910+0000 7f361b7fe640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f360806b840 0x7f360806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:35.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.910+0000 7f361b7fe640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f360806b840 0x7f360806dd00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f361c0ca230 tx=0x7f3610009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:35.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:35.913+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f360c07b400 con 0x7f361c06c0b0 2024-03-23T15:47:36.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.164+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f360c0776a0 con 0x7f361c06c0b0 2024-03-23T15:47:36.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.344+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f361c0c5800 con 0x7f360806b840 2024-03-23T15:47:36.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.350+0000 7f3618ff9640 1 -- 172.21.15.129:0/2590290923 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252009 (secure 0 0 0) 0x7f3610011a30 con 0x7f360806b840 2024-03-23T15:47:36.358 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:36.360 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T15:47:36.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f360806b840 msgr2=0x7f360806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:36.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f360806b840 0x7f360806dd00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f361c0ca230 tx=0x7f3610009190 comp rx=0 tx=0).stop 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f361c06c0b0 msgr2=0x7f361c1efb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f361c06c0b0 0x7f361c1efb20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f360c033000 tx=0x7f360c004730 comp rx=0 tx=0).stop 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 shutdown_connections 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f361c1544b0 0x7f361c1574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f361c0caa00 0x7f361c1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f360806b840 0x7f360806dd00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 --2- 172.21.15.129:0/2590290923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f361c06c0b0 0x7f361c1efb20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 >> 172.21.15.129:0/2590290923 conn(0x7f361c061b50 msgr2=0x7f361c0c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:36.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 shutdown_connections 2024-03-23T15:47:36.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.354+0000 7f36226c2640 1 -- 172.21.15.129:0/2590290923 wait complete. 2024-03-23T15:47:36.368 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-23T15:47:34.407065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230560,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754744160,"statfs":{"total":773094113280,"available":772858019840,"internally_reserved":0,"allocated":1933312,"data_stored":1793484,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001781"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686665+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:47:28.686665+0000","last_peered":"2024-03-23T15:47:28.686665+0000","last_clean":"2024-03-23T15:47:28.686665+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:47:28.686665+0000","last_undegraded":"2024-03-23T15:47:28.686665+0000","last_fullsized":"2024-03-23T15:47:28.686665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411203+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:47:28.411203+0000","last_peered":"2024-03-23T15:47:28.411203+0000","last_clean":"2024-03-23T15:47:28.411203+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:47:28.411203+0000","last_undegraded":"2024-03-23T15:47:28.411203+0000","last_fullsized":"2024-03-23T15:47:28.411203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411144+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:47:28.411144+0000","last_peered":"2024-03-23T15:47:28.411144+0000","last_clean":"2024-03-23T15:47:28.411144+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:47:28.411144+0000","last_undegraded":"2024-03-23T15:47:28.411144+0000","last_fullsized":"2024-03-23T15:47:28.411144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682515+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:47:28.682515+0000","last_peered":"2024-03-23T15:47:28.682515+0000","last_clean":"2024-03-23T15:47:28.682515+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:47:28.682515+0000","last_undegraded":"2024-03-23T15:47:28.682515+0000","last_fullsized":"2024-03-23T15:47:28.682515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686424+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:47:28.686424+0000","last_peered":"2024-03-23T15:47:28.686424+0000","last_clean":"2024-03-23T15:47:28.686424+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:47:28.686424+0000","last_undegraded":"2024-03-23T15:47:28.686424+0000","last_fullsized":"2024-03-23T15:47:28.686424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682361+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:47:28.682361+0000","last_peered":"2024-03-23T15:47:28.682361+0000","last_clean":"2024-03-23T15:47:28.682361+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:47:28.682361+0000","last_undegraded":"2024-03-23T15:47:28.682361+0000","last_fullsized":"2024-03-23T15:47:28.682361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686125+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:47:28.686125+0000","last_peered":"2024-03-23T15:47:28.686125+0000","last_clean":"2024-03-23T15:47:28.686125+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:47:28.686125+0000","last_undegraded":"2024-03-23T15:47:28.686125+0000","last_fullsized":"2024-03-23T15:47:28.686125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410933+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:47:28.410933+0000","last_peered":"2024-03-23T15:47:28.410933+0000","last_clean":"2024-03-23T15:47:28.410933+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:47:28.410933+0000","last_undegraded":"2024-03-23T15:47:28.410933+0000","last_fullsized":"2024-03-23T15:47:28.410933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465785+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:47:28.465785+0000","last_peered":"2024-03-23T15:47:28.465785+0000","last_clean":"2024-03-23T15:47:28.465785+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:47:28.465785+0000","last_undegraded":"2024-03-23T15:47:28.465785+0000","last_fullsized":"2024-03-23T15:47:28.465785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686368+0000","last_change":"2024-03-23T15:47:27.453034+0000","last_active":"2024-03-23T15:47:28.686368+0000","last_peered":"2024-03-23T15:47:28.686368+0000","last_clean":"2024-03-23T15:47:28.686368+0000","last_became_active":"2024-03-23T15:47:27.452786+0000","last_became_peered":"2024-03-23T15:47:27.452786+0000","last_unstale":"2024-03-23T15:47:28.686368+0000","last_undegraded":"2024-03-23T15:47:28.686368+0000","last_fullsized":"2024-03-23T15:47:28.686368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747756+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:47:28.747756+0000","last_peered":"2024-03-23T15:47:28.747756+0000","last_clean":"2024-03-23T15:47:28.747756+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:47:28.747756+0000","last_undegraded":"2024-03-23T15:47:28.747756+0000","last_fullsized":"2024-03-23T15:47:28.747756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747609+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:47:28.747609+0000","last_peered":"2024-03-23T15:47:28.747609+0000","last_clean":"2024-03-23T15:47:28.747609+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:47:28.747609+0000","last_undegraded":"2024-03-23T15:47:28.747609+0000","last_fullsized":"2024-03-23T15:47:28.747609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410854+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:47:28.410854+0000","last_peered":"2024-03-23T15:47:28.410854+0000","last_clean":"2024-03-23T15:47:28.410854+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:47:28.410854+0000","last_undegraded":"2024-03-23T15:47:28.410854+0000","last_fullsized":"2024-03-23T15:47:28.410854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416608+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:47:28.416608+0000","last_peered":"2024-03-23T15:47:28.416608+0000","last_clean":"2024-03-23T15:47:28.416608+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:47:28.416608+0000","last_undegraded":"2024-03-23T15:47:28.416608+0000","last_fullsized":"2024-03-23T15:47:28.416608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417524+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:47:28.417524+0000","last_peered":"2024-03-23T15:47:28.417524+0000","last_clean":"2024-03-23T15:47:28.417524+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:47:28.417524+0000","last_undegraded":"2024-03-23T15:47:28.417524+0000","last_fullsized":"2024-03-23T15:47:28.417524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681899+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:47:28.681899+0000","last_peered":"2024-03-23T15:47:28.681899+0000","last_clean":"2024-03-23T15:47:28.681899+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:47:28.681899+0000","last_undegraded":"2024-03-23T15:47:28.681899+0000","last_fullsized":"2024-03-23T15:47:28.681899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417419+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:47:28.417419+0000","last_peered":"2024-03-23T15:47:28.417419+0000","last_clean":"2024-03-23T15:47:28.417419+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:47:28.417419+0000","last_undegraded":"2024-03-23T15:47:28.417419+0000","last_fullsized":"2024-03-23T15:47:28.417419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686539+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:47:28.686539+0000","last_peered":"2024-03-23T15:47:28.686539+0000","last_clean":"2024-03-23T15:47:28.686539+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:47:28.686539+0000","last_undegraded":"2024-03-23T15:47:28.686539+0000","last_fullsized":"2024-03-23T15:47:28.686539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679986+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:47:28.679986+0000","last_peered":"2024-03-23T15:47:28.679986+0000","last_clean":"2024-03-23T15:47:28.679986+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:47:28.679986+0000","last_undegraded":"2024-03-23T15:47:28.679986+0000","last_fullsized":"2024-03-23T15:47:28.679986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465648+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:47:28.465648+0000","last_peered":"2024-03-23T15:47:28.465648+0000","last_clean":"2024-03-23T15:47:28.465648+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:47:28.465648+0000","last_undegraded":"2024-03-23T15:47:28.465648+0000","last_fullsized":"2024-03-23T15:47:28.465648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410906+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:47:28.410906+0000","last_peered":"2024-03-23T15:47:28.410906+0000","last_clean":"2024-03-23T15:47:28.410906+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:47:28.410906+0000","last_undegraded":"2024-03-23T15:47:28.410906+0000","last_fullsized":"2024-03-23T15:47:28.410906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416609+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:47:28.416609+0000","last_peered":"2024-03-23T15:47:28.416609+0000","last_clean":"2024-03-23T15:47:28.416609+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:47:28.416609+0000","last_undegraded":"2024-03-23T15:47:28.416609+0000","last_fullsized":"2024-03-23T15:47:28.416609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411310+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:47:28.411310+0000","last_peered":"2024-03-23T15:47:28.411310+0000","last_clean":"2024-03-23T15:47:28.411310+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:47:28.411310+0000","last_undegraded":"2024-03-23T15:47:28.411310+0000","last_fullsized":"2024-03-23T15:47:28.411310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682742+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:47:28.682742+0000","last_peered":"2024-03-23T15:47:28.682742+0000","last_clean":"2024-03-23T15:47:28.682742+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:47:28.682742+0000","last_undegraded":"2024-03-23T15:47:28.682742+0000","last_fullsized":"2024-03-23T15:47:28.682742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747920+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:47:28.747920+0000","last_peered":"2024-03-23T15:47:28.747920+0000","last_clean":"2024-03-23T15:47:28.747920+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:47:28.747920+0000","last_undegraded":"2024-03-23T15:47:28.747920+0000","last_fullsized":"2024-03-23T15:47:28.747920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747807+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:47:28.747807+0000","last_peered":"2024-03-23T15:47:28.747807+0000","last_clean":"2024-03-23T15:47:28.747807+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:47:28.747807+0000","last_undegraded":"2024-03-23T15:47:28.747807+0000","last_fullsized":"2024-03-23T15:47:28.747807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411018+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:47:28.411018+0000","last_peered":"2024-03-23T15:47:28.411018+0000","last_clean":"2024-03-23T15:47:28.411018+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:47:28.411018+0000","last_undegraded":"2024-03-23T15:47:28.411018+0000","last_fullsized":"2024-03-23T15:47:28.411018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416777+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:47:28.416777+0000","last_peered":"2024-03-23T15:47:28.416777+0000","last_clean":"2024-03-23T15:47:28.416777+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:47:28.416777+0000","last_undegraded":"2024-03-23T15:47:28.416777+0000","last_fullsized":"2024-03-23T15:47:28.416777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681919+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:47:28.681919+0000","last_peered":"2024-03-23T15:47:28.681919+0000","last_clean":"2024-03-23T15:47:28.681919+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:47:28.681919+0000","last_undegraded":"2024-03-23T15:47:28.681919+0000","last_fullsized":"2024-03-23T15:47:28.681919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682193+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:47:28.682193+0000","last_peered":"2024-03-23T15:47:28.682193+0000","last_clean":"2024-03-23T15:47:28.682193+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:47:28.682193+0000","last_undegraded":"2024-03-23T15:47:28.682193+0000","last_fullsized":"2024-03-23T15:47:28.682193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465791+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:47:28.465791+0000","last_peered":"2024-03-23T15:47:28.465791+0000","last_clean":"2024-03-23T15:47:28.465791+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:47:28.465791+0000","last_undegraded":"2024-03-23T15:47:28.465791+0000","last_fullsized":"2024-03-23T15:47:28.465791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411177+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:47:28.411177+0000","last_peered":"2024-03-23T15:47:28.411177+0000","last_clean":"2024-03-23T15:47:28.411177+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:47:28.411177+0000","last_undegraded":"2024-03-23T15:47:28.411177+0000","last_fullsized":"2024-03-23T15:47:28.411177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680247+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:47:28.680247+0000","last_peered":"2024-03-23T15:47:28.680247+0000","last_clean":"2024-03-23T15:47:28.680247+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:47:28.680247+0000","last_undegraded":"2024-03-23T15:47:28.680247+0000","last_fullsized":"2024-03-23T15:47:28.680247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417004+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:47:28.417004+0000","last_peered":"2024-03-23T15:47:28.417004+0000","last_clean":"2024-03-23T15:47:28.417004+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:47:28.417004+0000","last_undegraded":"2024-03-23T15:47:28.417004+0000","last_fullsized":"2024-03-23T15:47:28.417004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747536+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:47:28.747536+0000","last_peered":"2024-03-23T15:47:28.747536+0000","last_clean":"2024-03-23T15:47:28.747536+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:47:28.747536+0000","last_undegraded":"2024-03-23T15:47:28.747536+0000","last_fullsized":"2024-03-23T15:47:28.747536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411290+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:47:28.411290+0000","last_peered":"2024-03-23T15:47:28.411290+0000","last_clean":"2024-03-23T15:47:28.411290+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:47:28.411290+0000","last_undegraded":"2024-03-23T15:47:28.411290+0000","last_fullsized":"2024-03-23T15:47:28.411290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416713+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:47:28.416713+0000","last_peered":"2024-03-23T15:47:28.416713+0000","last_clean":"2024-03-23T15:47:28.416713+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:47:28.416713+0000","last_undegraded":"2024-03-23T15:47:28.416713+0000","last_fullsized":"2024-03-23T15:47:28.416713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679735+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:47:28.679735+0000","last_peered":"2024-03-23T15:47:28.679735+0000","last_clean":"2024-03-23T15:47:28.679735+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:47:28.679735+0000","last_undegraded":"2024-03-23T15:47:28.679735+0000","last_fullsized":"2024-03-23T15:47:28.679735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465958+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:47:28.465958+0000","last_peered":"2024-03-23T15:47:28.465958+0000","last_clean":"2024-03-23T15:47:28.465958+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:47:28.465958+0000","last_undegraded":"2024-03-23T15:47:28.465958+0000","last_fullsized":"2024-03-23T15:47:28.465958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682790+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:47:28.682790+0000","last_peered":"2024-03-23T15:47:28.682790+0000","last_clean":"2024-03-23T15:47:28.682790+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:47:28.682790+0000","last_undegraded":"2024-03-23T15:47:28.682790+0000","last_fullsized":"2024-03-23T15:47:28.682790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686452+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:47:28.686452+0000","last_peered":"2024-03-23T15:47:28.686452+0000","last_clean":"2024-03-23T15:47:28.686452+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:47:28.686452+0000","last_undegraded":"2024-03-23T15:47:28.686452+0000","last_fullsized":"2024-03-23T15:47:28.686452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411427+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:47:28.411427+0000","last_peered":"2024-03-23T15:47:28.411427+0000","last_clean":"2024-03-23T15:47:28.411427+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:47:28.411427+0000","last_undegraded":"2024-03-23T15:47:28.411427+0000","last_fullsized":"2024-03-23T15:47:28.411427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.684821+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:47:28.684821+0000","last_peered":"2024-03-23T15:47:28.684821+0000","last_clean":"2024-03-23T15:47:28.684821+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:47:28.684821+0000","last_undegraded":"2024-03-23T15:47:28.684821+0000","last_fullsized":"2024-03-23T15:47:28.684821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416762+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:47:28.416762+0000","last_peered":"2024-03-23T15:47:28.416762+0000","last_clean":"2024-03-23T15:47:28.416762+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:47:28.416762+0000","last_undegraded":"2024-03-23T15:47:28.416762+0000","last_fullsized":"2024-03-23T15:47:28.416762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.684784+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:47:28.684784+0000","last_peered":"2024-03-23T15:47:28.684784+0000","last_clean":"2024-03-23T15:47:28.684784+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:47:28.684784+0000","last_undegraded":"2024-03-23T15:47:28.684784+0000","last_fullsized":"2024-03-23T15:47:28.684784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686519+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:47:28.686519+0000","last_peered":"2024-03-23T15:47:28.686519+0000","last_clean":"2024-03-23T15:47:28.686519+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:47:28.686519+0000","last_undegraded":"2024-03-23T15:47:28.686519+0000","last_fullsized":"2024-03-23T15:47:28.686519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.685991+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:47:28.685991+0000","last_peered":"2024-03-23T15:47:28.685991+0000","last_clean":"2024-03-23T15:47:28.685991+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:47:28.685991+0000","last_undegraded":"2024-03-23T15:47:28.685991+0000","last_fullsized":"2024-03-23T15:47:28.685991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747624+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:47:28.747624+0000","last_peered":"2024-03-23T15:47:28.747624+0000","last_clean":"2024-03-23T15:47:28.747624+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:47:28.747624+0000","last_undegraded":"2024-03-23T15:47:28.747624+0000","last_fullsized":"2024-03-23T15:47:28.747624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417028+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:47:28.417028+0000","last_peered":"2024-03-23T15:47:28.417028+0000","last_clean":"2024-03-23T15:47:28.417028+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:47:28.417028+0000","last_undegraded":"2024-03-23T15:47:28.417028+0000","last_fullsized":"2024-03-23T15:47:28.417028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416928+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:47:28.416928+0000","last_peered":"2024-03-23T15:47:28.416928+0000","last_clean":"2024-03-23T15:47:28.416928+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:47:28.416928+0000","last_undegraded":"2024-03-23T15:47:28.416928+0000","last_fullsized":"2024-03-23T15:47:28.416928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686074+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:47:28.686074+0000","last_peered":"2024-03-23T15:47:28.686074+0000","last_clean":"2024-03-23T15:47:28.686074+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:47:28.686074+0000","last_undegraded":"2024-03-23T15:47:28.686074+0000","last_fullsized":"2024-03-23T15:47:28.686074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465425+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:47:28.465425+0000","last_peered":"2024-03-23T15:47:28.465425+0000","last_clean":"2024-03-23T15:47:28.465425+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:47:28.465425+0000","last_undegraded":"2024-03-23T15:47:28.465425+0000","last_fullsized":"2024-03-23T15:47:28.465425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.684701+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:47:28.684701+0000","last_peered":"2024-03-23T15:47:28.684701+0000","last_clean":"2024-03-23T15:47:28.684701+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:47:28.684701+0000","last_undegraded":"2024-03-23T15:47:28.684701+0000","last_fullsized":"2024-03-23T15:47:28.684701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416623+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:47:28.416623+0000","last_peered":"2024-03-23T15:47:28.416623+0000","last_clean":"2024-03-23T15:47:28.416623+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:47:28.416623+0000","last_undegraded":"2024-03-23T15:47:28.416623+0000","last_fullsized":"2024-03-23T15:47:28.416623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686786+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:47:28.686786+0000","last_peered":"2024-03-23T15:47:28.686786+0000","last_clean":"2024-03-23T15:47:28.686786+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:47:28.686786+0000","last_undegraded":"2024-03-23T15:47:28.686786+0000","last_fullsized":"2024-03-23T15:47:28.686786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682875+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:47:28.682875+0000","last_peered":"2024-03-23T15:47:28.682875+0000","last_clean":"2024-03-23T15:47:28.682875+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:47:28.682875+0000","last_undegraded":"2024-03-23T15:47:28.682875+0000","last_fullsized":"2024-03-23T15:47:28.682875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.683735+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:47:28.683735+0000","last_peered":"2024-03-23T15:47:28.683735+0000","last_clean":"2024-03-23T15:47:28.683735+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:47:28.683735+0000","last_undegraded":"2024-03-23T15:47:28.683735+0000","last_fullsized":"2024-03-23T15:47:28.683735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686676+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:47:28.686676+0000","last_peered":"2024-03-23T15:47:28.686676+0000","last_clean":"2024-03-23T15:47:28.686676+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:47:28.686676+0000","last_undegraded":"2024-03-23T15:47:28.686676+0000","last_fullsized":"2024-03-23T15:47:28.686676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411312+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:47:28.411312+0000","last_peered":"2024-03-23T15:47:28.411312+0000","last_clean":"2024-03-23T15:47:28.411312+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:47:28.411312+0000","last_undegraded":"2024-03-23T15:47:28.411312+0000","last_fullsized":"2024-03-23T15:47:28.411312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686633+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:47:28.686633+0000","last_peered":"2024-03-23T15:47:28.686633+0000","last_clean":"2024-03-23T15:47:28.686633+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:47:28.686633+0000","last_undegraded":"2024-03-23T15:47:28.686633+0000","last_fullsized":"2024-03-23T15:47:28.686633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747921+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:47:28.747921+0000","last_peered":"2024-03-23T15:47:28.747921+0000","last_clean":"2024-03-23T15:47:28.747921+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:47:28.747921+0000","last_undegraded":"2024-03-23T15:47:28.747921+0000","last_fullsized":"2024-03-23T15:47:28.747921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417319+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:47:28.417319+0000","last_peered":"2024-03-23T15:47:28.417319+0000","last_clean":"2024-03-23T15:47:28.417319+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:47:28.417319+0000","last_undegraded":"2024-03-23T15:47:28.417319+0000","last_fullsized":"2024-03-23T15:47:28.417319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":171,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465446+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:47:28.465446+0000","last_peered":"2024-03-23T15:47:28.465446+0000","last_clean":"2024-03-23T15:47:28.465446+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:47:28.465446+0000","last_undegraded":"2024-03-23T15:47:28.465446+0000","last_fullsized":"2024-03-23T15:47:28.465446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417123+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:47:28.417123+0000","last_peered":"2024-03-23T15:47:28.417123+0000","last_clean":"2024-03-23T15:47:28.417123+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:47:28.417123+0000","last_undegraded":"2024-03-23T15:47:28.417123+0000","last_fullsized":"2024-03-23T15:47:28.417123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679960+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:47:28.679960+0000","last_peered":"2024-03-23T15:47:28.679960+0000","last_clean":"2024-03-23T15:47:28.679960+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:47:28.679960+0000","last_undegraded":"2024-03-23T15:47:28.679960+0000","last_fullsized":"2024-03-23T15:47:28.679960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416855+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:47:28.416855+0000","last_peered":"2024-03-23T15:47:28.416855+0000","last_clean":"2024-03-23T15:47:28.416855+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:47:28.416855+0000","last_undegraded":"2024-03-23T15:47:28.416855+0000","last_fullsized":"2024-03-23T15:47:28.416855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747588+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:47:28.747588+0000","last_peered":"2024-03-23T15:47:28.747588+0000","last_clean":"2024-03-23T15:47:28.747588+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:47:28.747588+0000","last_undegraded":"2024-03-23T15:47:28.747588+0000","last_fullsized":"2024-03-23T15:47:28.747588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682011+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:47:28.682011+0000","last_peered":"2024-03-23T15:47:28.682011+0000","last_clean":"2024-03-23T15:47:28.682011+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:47:28.682011+0000","last_undegraded":"2024-03-23T15:47:28.682011+0000","last_fullsized":"2024-03-23T15:47:28.682011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411099+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:47:28.411099+0000","last_peered":"2024-03-23T15:47:28.411099+0000","last_clean":"2024-03-23T15:47:28.411099+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:47:28.411099+0000","last_undegraded":"2024-03-23T15:47:28.411099+0000","last_fullsized":"2024-03-23T15:47:28.411099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417084+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:47:28.417084+0000","last_peered":"2024-03-23T15:47:28.417084+0000","last_clean":"2024-03-23T15:47:28.417084+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:47:28.417084+0000","last_undegraded":"2024-03-23T15:47:28.417084+0000","last_fullsized":"2024-03-23T15:47:28.417084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686285+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:47:28.686285+0000","last_peered":"2024-03-23T15:47:28.686285+0000","last_clean":"2024-03-23T15:47:28.686285+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:47:28.686285+0000","last_undegraded":"2024-03-23T15:47:28.686285+0000","last_fullsized":"2024-03-23T15:47:28.686285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411034+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:47:28.411034+0000","last_peered":"2024-03-23T15:47:28.411034+0000","last_clean":"2024-03-23T15:47:28.411034+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:47:28.411034+0000","last_undegraded":"2024-03-23T15:47:28.411034+0000","last_fullsized":"2024-03-23T15:47:28.411034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682390+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:47:28.682390+0000","last_peered":"2024-03-23T15:47:28.682390+0000","last_clean":"2024-03-23T15:47:28.682390+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:47:28.682390+0000","last_undegraded":"2024-03-23T15:47:28.682390+0000","last_fullsized":"2024-03-23T15:47:28.682390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417398+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:47:28.417398+0000","last_peered":"2024-03-23T15:47:28.417398+0000","last_clean":"2024-03-23T15:47:28.417398+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:47:28.417398+0000","last_undegraded":"2024-03-23T15:47:28.417398+0000","last_fullsized":"2024-03-23T15:47:28.417398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747808+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:47:28.747808+0000","last_peered":"2024-03-23T15:47:28.747808+0000","last_clean":"2024-03-23T15:47:28.747808+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:47:28.747808+0000","last_undegraded":"2024-03-23T15:47:28.747808+0000","last_fullsized":"2024-03-23T15:47:28.747808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680340+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:47:28.680340+0000","last_peered":"2024-03-23T15:47:28.680340+0000","last_clean":"2024-03-23T15:47:28.680340+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:47:28.680340+0000","last_undegraded":"2024-03-23T15:47:28.680340+0000","last_fullsized":"2024-03-23T15:47:28.680340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680240+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:47:28.680240+0000","last_peered":"2024-03-23T15:47:28.680240+0000","last_clean":"2024-03-23T15:47:28.680240+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:47:28.680240+0000","last_undegraded":"2024-03-23T15:47:28.680240+0000","last_fullsized":"2024-03-23T15:47:28.680240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.794912+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:47:28.794912+0000","last_peered":"2024-03-23T15:47:28.794912+0000","last_clean":"2024-03-23T15:47:28.794912+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:47:28.794912+0000","last_undegraded":"2024-03-23T15:47:28.794912+0000","last_fullsized":"2024-03-23T15:47:28.794912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680148+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:47:28.680148+0000","last_peered":"2024-03-23T15:47:28.680148+0000","last_clean":"2024-03-23T15:47:28.680148+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:47:28.680148+0000","last_undegraded":"2024-03-23T15:47:28.680148+0000","last_fullsized":"2024-03-23T15:47:28.680148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.685871+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:47:28.685871+0000","last_peered":"2024-03-23T15:47:28.685871+0000","last_clean":"2024-03-23T15:47:28.685871+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:47:28.685871+0000","last_undegraded":"2024-03-23T15:47:28.685871+0000","last_fullsized":"2024-03-23T15:47:28.685871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416764+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:47:28.416764+0000","last_peered":"2024-03-23T15:47:28.416764+0000","last_clean":"2024-03-23T15:47:28.416764+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:47:28.416764+0000","last_undegraded":"2024-03-23T15:47:28.416764+0000","last_fullsized":"2024-03-23T15:47:28.416764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679833+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:47:28.679833+0000","last_peered":"2024-03-23T15:47:28.679833+0000","last_clean":"2024-03-23T15:47:28.679833+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:47:28.679833+0000","last_undegraded":"2024-03-23T15:47:28.679833+0000","last_fullsized":"2024-03-23T15:47:28.679833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679874+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:47:28.679874+0000","last_peered":"2024-03-23T15:47:28.679874+0000","last_clean":"2024-03-23T15:47:28.679874+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:47:28.679874+0000","last_undegraded":"2024-03-23T15:47:28.679874+0000","last_fullsized":"2024-03-23T15:47:28.679874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747779+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:47:28.747779+0000","last_peered":"2024-03-23T15:47:28.747779+0000","last_clean":"2024-03-23T15:47:28.747779+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:47:28.747779+0000","last_undegraded":"2024-03-23T15:47:28.747779+0000","last_fullsized":"2024-03-23T15:47:28.747779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682112+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:47:28.682112+0000","last_peered":"2024-03-23T15:47:28.682112+0000","last_clean":"2024-03-23T15:47:28.682112+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:47:28.682112+0000","last_undegraded":"2024-03-23T15:47:28.682112+0000","last_fullsized":"2024-03-23T15:47:28.682112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.794901+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:47:28.794901+0000","last_peered":"2024-03-23T15:47:28.794901+0000","last_clean":"2024-03-23T15:47:28.794901+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:47:28.794901+0000","last_undegraded":"2024-03-23T15:47:28.794901+0000","last_fullsized":"2024-03-23T15:47:28.794901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416913+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:47:28.416913+0000","last_peered":"2024-03-23T15:47:28.416913+0000","last_clean":"2024-03-23T15:47:28.416913+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:47:28.416913+0000","last_undegraded":"2024-03-23T15:47:28.416913+0000","last_fullsized":"2024-03-23T15:47:28.416913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416753+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:47:28.416753+0000","last_peered":"2024-03-23T15:47:28.416753+0000","last_clean":"2024-03-23T15:47:28.416753+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:47:28.416753+0000","last_undegraded":"2024-03-23T15:47:28.416753+0000","last_fullsized":"2024-03-23T15:47:28.416753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686726+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:47:28.686726+0000","last_peered":"2024-03-23T15:47:28.686726+0000","last_clean":"2024-03-23T15:47:28.686726+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:47:28.686726+0000","last_undegraded":"2024-03-23T15:47:28.686726+0000","last_fullsized":"2024-03-23T15:47:28.686726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465591+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:47:28.465591+0000","last_peered":"2024-03-23T15:47:28.465591+0000","last_clean":"2024-03-23T15:47:28.465591+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:47:28.465591+0000","last_undegraded":"2024-03-23T15:47:28.465591+0000","last_fullsized":"2024-03-23T15:47:28.465591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686610+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:47:28.686610+0000","last_peered":"2024-03-23T15:47:28.686610+0000","last_clean":"2024-03-23T15:47:28.686610+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:47:28.686610+0000","last_undegraded":"2024-03-23T15:47:28.686610+0000","last_fullsized":"2024-03-23T15:47:28.686610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411323+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:47:28.411323+0000","last_peered":"2024-03-23T15:47:28.411323+0000","last_clean":"2024-03-23T15:47:28.411323+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:47:28.411323+0000","last_undegraded":"2024-03-23T15:47:28.411323+0000","last_fullsized":"2024-03-23T15:47:28.411323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411370+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:47:28.411370+0000","last_peered":"2024-03-23T15:47:28.411370+0000","last_clean":"2024-03-23T15:47:28.411370+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:47:28.411370+0000","last_undegraded":"2024-03-23T15:47:28.411370+0000","last_fullsized":"2024-03-23T15:47:28.411370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411212+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:47:28.411212+0000","last_peered":"2024-03-23T15:47:28.411212+0000","last_clean":"2024-03-23T15:47:28.411212+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:47:28.411212+0000","last_undegraded":"2024-03-23T15:47:28.411212+0000","last_fullsized":"2024-03-23T15:47:28.411212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.685789+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:47:28.685789+0000","last_peered":"2024-03-23T15:47:28.685789+0000","last_clean":"2024-03-23T15:47:28.685789+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:47:28.685789+0000","last_undegraded":"2024-03-23T15:47:28.685789+0000","last_fullsized":"2024-03-23T15:47:28.685789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31468,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340372,"statfs":{"total":96636764160,"available":96604540928,"internally_reserved":0,"allocated":290816,"data_stored":666451,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":237568,"data_stored":78275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":76573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":666429,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":76047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T15:47:36.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T15:47:36.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.510+0000 7f6703fd0640 1 Processor -- start 2024-03-23T15:47:36.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.510+0000 7f6703fd0640 1 -- start start 2024-03-23T15:47:36.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.511+0000 7f6703fd0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f66fc151320 0x7f66fc157a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.511+0000 7f6703fd0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc06b130 0x7f66fc072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.511+0000 7f6703fd0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f66fc073380 0x7f66fc0ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.511+0000 7f6703fd0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f66fc0c1480 con 0x7f66fc151320 2024-03-23T15:47:36.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.511+0000 7f6703fd0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f66fc0c1720 con 0x7f66fc073380 2024-03-23T15:47:36.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.511+0000 7f6703fd0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f66fc0c1a00 con 0x7f66fc06b130 2024-03-23T15:47:36.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.520+0000 7f6701d6c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f66fc151320 0x7f66fc157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:36.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.520+0000 7f6701d6c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f66fc151320 0x7f66fc157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58812/0 (socket says 172.21.15.129:58812) 2024-03-23T15:47:36.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.520+0000 7f6701d6c640 1 -- 172.21.15.129:0/1242536689 learned_addr learned my addr 172.21.15.129:0/1242536689 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:36.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.522+0000 7f6701d6c640 1 -- 172.21.15.129:0/1242536689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc06b130 msgr2=0x7f66fc072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:36.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.522+0000 7f6701d6c640 1 --2- 172.21.15.129:0/1242536689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc06b130 0x7f66fc072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.522+0000 7f6701d6c640 1 -- 172.21.15.129:0/1242536689 >> v2:172.21.15.157:3300/0 conn(0x7f66fc073380 msgr2=0x7f66fc0ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:36.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.522+0000 7f6701d6c640 1 --2- 172.21.15.129:0/1242536689 >> v2:172.21.15.157:3300/0 conn(0x7f66fc073380 0x7f66fc0ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.522+0000 7f6701d6c640 1 -- 172.21.15.129:0/1242536689 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc157260 con 0x7f66fc151320 2024-03-23T15:47:36.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.525+0000 7f6701d6c640 1 --2- 172.21.15.129:0/1242536689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc151320 0x7f66fc157a60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66fc06cf30 tx=0x7f66f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2356f2f5b6d304 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.528+0000 7f6700d6a640 1 -- 172.21.15.129:0/1242536689 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8031870 con 0x7f66fc151320 2024-03-23T15:47:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.528+0000 7f6700d6a640 1 -- 172.21.15.129:0/1242536689 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f80319d0 con 0x7f66fc151320 2024-03-23T15:47:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.528+0000 7f6700d6a640 1 -- 172.21.15.129:0/1242536689 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8031d60 con 0x7f66fc151320 2024-03-23T15:47:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 -- 172.21.15.129:0/1242536689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc151320 msgr2=0x7f66fc157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1242536689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc151320 0x7f66fc157a60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66fc06cf30 tx=0x7f66f802b910 comp rx=0 tx=0).stop 2024-03-23T15:47:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 -- 172.21.15.129:0/1242536689 shutdown_connections 2024-03-23T15:47:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1242536689 >> v2:172.21.15.157:3300/0 conn(0x7f66fc073380 0x7f66fc0ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1242536689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc06b130 0x7f66fc072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1242536689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc151320 0x7f66fc157a60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 -- 172.21.15.129:0/1242536689 >> 172.21.15.129:0/1242536689 conn(0x7f66fc0cc120 msgr2=0x7f66fc0cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 -- 172.21.15.129:0/1242536689 shutdown_connections 2024-03-23T15:47:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.529+0000 7f6703fd0640 1 -- 172.21.15.129:0/1242536689 wait complete. 2024-03-23T15:47:36.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.531+0000 7f6703fd0640 1 Processor -- start 2024-03-23T15:47:36.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.531+0000 7f6703fd0640 1 -- start start 2024-03-23T15:47:36.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.532+0000 7f6703fd0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66fc06b130 0x7f66fc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f6703fd0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 0x7f66fc0c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f6703fd0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc151320 0x7f66fc151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f6703fd0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc0c55d0 con 0x7f66fc073380 2024-03-23T15:47:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f6703fd0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc0c4590 con 0x7f66fc06b130 2024-03-23T15:47:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f6703fd0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f66fc0c67c0 con 0x7f66fc151320 2024-03-23T15:47:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f670156b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 0x7f66fc0c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f670156b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 0x7f66fc0c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58816/0 (socket says 172.21.15.129:58816) 2024-03-23T15:47:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.533+0000 7f670156b640 1 -- 172.21.15.129:0/1497575351 learned_addr learned my addr 172.21.15.129:0/1497575351 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:36.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.537+0000 7f670156b640 1 -- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc151320 msgr2=0x7f66fc151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:36.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.537+0000 7f670156b640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc151320 0x7f66fc151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.537+0000 7f670156b640 1 -- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66fc06b130 msgr2=0x7f66fc156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.537+0000 7f670156b640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66fc06b130 0x7f66fc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.537+0000 7f670156b640 1 -- 172.21.15.129:0/1497575351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f80094d0 con 0x7f66fc073380 2024-03-23T15:47:36.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.539+0000 7f670156b640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 0x7f66fc0c5e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00b4a0 tx=0x7f66ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:36.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.542+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec010610 con 0x7f66fc073380 2024-03-23T15:47:36.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.542+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66ec010770 con 0x7f66fc073380 2024-03-23T15:47:36.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.542+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66fc0c6aa0 con 0x7f66fc073380 2024-03-23T15:47:36.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.542+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec010b20 con 0x7f66fc073380 2024-03-23T15:47:36.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.543+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66fc0c3ed0 con 0x7f66fc073380 2024-03-23T15:47:36.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.543+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66fc075740 con 0x7f66fc073380 2024-03-23T15:47:36.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.861+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66ec010c80 con 0x7f66fc073380 2024-03-23T15:47:36.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.861+0000 7f66f27fc640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f66e006b810 0x7f66e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:36.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.862+0000 7f6701d6c640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f66e006b810 0x7f66e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:36.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.862+0000 7f6701d6c640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f66e006b810 0x7f66e006dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66f8036000 tx=0x7f66f8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:36.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.892+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f66ec059140 con 0x7f66fc073380 2024-03-23T15:47:36.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:36.892+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66ec054ce0 con 0x7f66fc073380 2024-03-23T15:47:37.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.077+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66fc157fa0 con 0x7f66e006b810 2024-03-23T15:47:37.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.090+0000 7f66f27fc640 1 -- 172.21.15.129:0/1497575351 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252028 (secure 0 0 0) 0x7f66f8031bc0 con 0x7f66e006b810 2024-03-23T15:47:37.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:37.097 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T15:47:37.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f66e006b810 msgr2=0x7f66e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f66e006b810 0x7f66e006dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66f8036000 tx=0x7f66f8030000 comp rx=0 tx=0).stop 2024-03-23T15:47:37.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 msgr2=0x7f66fc0c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 0x7f66fc0c5e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00b4a0 tx=0x7f66ec00b960 comp rx=0 tx=0).stop 2024-03-23T15:47:37.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 shutdown_connections 2024-03-23T15:47:37.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66fc151320 0x7f66fc151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66fc073380 0x7f66fc0c5e80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f66e006b810 0x7f66e006dcd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 --2- 172.21.15.129:0/1497575351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66fc06b130 0x7f66fc156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 >> 172.21.15.129:0/1497575351 conn(0x7f66fc0cc120 msgr2=0x7f66fc0c7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 shutdown_connections 2024-03-23T15:47:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.093+0000 7f6703fd0640 1 -- 172.21.15.129:0/1497575351 wait complete. 2024-03-23T15:47:37.108 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-23T15:47:36.407465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":282,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230592,"kb_used_data":1984,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754744128,"statfs":{"total":773094113280,"available":772857987072,"internally_reserved":0,"allocated":2031616,"data_stored":1799108,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5222,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.058983"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686665+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:47:28.686665+0000","last_peered":"2024-03-23T15:47:28.686665+0000","last_clean":"2024-03-23T15:47:28.686665+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:47:28.686665+0000","last_undegraded":"2024-03-23T15:47:28.686665+0000","last_fullsized":"2024-03-23T15:47:28.686665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411203+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:47:28.411203+0000","last_peered":"2024-03-23T15:47:28.411203+0000","last_clean":"2024-03-23T15:47:28.411203+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:47:28.411203+0000","last_undegraded":"2024-03-23T15:47:28.411203+0000","last_fullsized":"2024-03-23T15:47:28.411203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411144+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:47:28.411144+0000","last_peered":"2024-03-23T15:47:28.411144+0000","last_clean":"2024-03-23T15:47:28.411144+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:47:28.411144+0000","last_undegraded":"2024-03-23T15:47:28.411144+0000","last_fullsized":"2024-03-23T15:47:28.411144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682515+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:47:28.682515+0000","last_peered":"2024-03-23T15:47:28.682515+0000","last_clean":"2024-03-23T15:47:28.682515+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:47:28.682515+0000","last_undegraded":"2024-03-23T15:47:28.682515+0000","last_fullsized":"2024-03-23T15:47:28.682515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686424+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:47:28.686424+0000","last_peered":"2024-03-23T15:47:28.686424+0000","last_clean":"2024-03-23T15:47:28.686424+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:47:28.686424+0000","last_undegraded":"2024-03-23T15:47:28.686424+0000","last_fullsized":"2024-03-23T15:47:28.686424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682361+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:47:28.682361+0000","last_peered":"2024-03-23T15:47:28.682361+0000","last_clean":"2024-03-23T15:47:28.682361+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:47:28.682361+0000","last_undegraded":"2024-03-23T15:47:28.682361+0000","last_fullsized":"2024-03-23T15:47:28.682361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686125+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:47:28.686125+0000","last_peered":"2024-03-23T15:47:28.686125+0000","last_clean":"2024-03-23T15:47:28.686125+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:47:28.686125+0000","last_undegraded":"2024-03-23T15:47:28.686125+0000","last_fullsized":"2024-03-23T15:47:28.686125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410933+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:47:28.410933+0000","last_peered":"2024-03-23T15:47:28.410933+0000","last_clean":"2024-03-23T15:47:28.410933+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:47:28.410933+0000","last_undegraded":"2024-03-23T15:47:28.410933+0000","last_fullsized":"2024-03-23T15:47:28.410933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465785+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:47:28.465785+0000","last_peered":"2024-03-23T15:47:28.465785+0000","last_clean":"2024-03-23T15:47:28.465785+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:47:28.465785+0000","last_undegraded":"2024-03-23T15:47:28.465785+0000","last_fullsized":"2024-03-23T15:47:28.465785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686368+0000","last_change":"2024-03-23T15:47:27.453034+0000","last_active":"2024-03-23T15:47:28.686368+0000","last_peered":"2024-03-23T15:47:28.686368+0000","last_clean":"2024-03-23T15:47:28.686368+0000","last_became_active":"2024-03-23T15:47:27.452786+0000","last_became_peered":"2024-03-23T15:47:27.452786+0000","last_unstale":"2024-03-23T15:47:28.686368+0000","last_undegraded":"2024-03-23T15:47:28.686368+0000","last_fullsized":"2024-03-23T15:47:28.686368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747756+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:47:28.747756+0000","last_peered":"2024-03-23T15:47:28.747756+0000","last_clean":"2024-03-23T15:47:28.747756+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:47:28.747756+0000","last_undegraded":"2024-03-23T15:47:28.747756+0000","last_fullsized":"2024-03-23T15:47:28.747756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747609+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:47:28.747609+0000","last_peered":"2024-03-23T15:47:28.747609+0000","last_clean":"2024-03-23T15:47:28.747609+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:47:28.747609+0000","last_undegraded":"2024-03-23T15:47:28.747609+0000","last_fullsized":"2024-03-23T15:47:28.747609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410854+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:47:28.410854+0000","last_peered":"2024-03-23T15:47:28.410854+0000","last_clean":"2024-03-23T15:47:28.410854+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:47:28.410854+0000","last_undegraded":"2024-03-23T15:47:28.410854+0000","last_fullsized":"2024-03-23T15:47:28.410854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416608+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:47:28.416608+0000","last_peered":"2024-03-23T15:47:28.416608+0000","last_clean":"2024-03-23T15:47:28.416608+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:47:28.416608+0000","last_undegraded":"2024-03-23T15:47:28.416608+0000","last_fullsized":"2024-03-23T15:47:28.416608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417524+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:47:28.417524+0000","last_peered":"2024-03-23T15:47:28.417524+0000","last_clean":"2024-03-23T15:47:28.417524+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:47:28.417524+0000","last_undegraded":"2024-03-23T15:47:28.417524+0000","last_fullsized":"2024-03-23T15:47:28.417524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681899+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:47:28.681899+0000","last_peered":"2024-03-23T15:47:28.681899+0000","last_clean":"2024-03-23T15:47:28.681899+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:47:28.681899+0000","last_undegraded":"2024-03-23T15:47:28.681899+0000","last_fullsized":"2024-03-23T15:47:28.681899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417419+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:47:28.417419+0000","last_peered":"2024-03-23T15:47:28.417419+0000","last_clean":"2024-03-23T15:47:28.417419+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:47:28.417419+0000","last_undegraded":"2024-03-23T15:47:28.417419+0000","last_fullsized":"2024-03-23T15:47:28.417419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686539+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:47:28.686539+0000","last_peered":"2024-03-23T15:47:28.686539+0000","last_clean":"2024-03-23T15:47:28.686539+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:47:28.686539+0000","last_undegraded":"2024-03-23T15:47:28.686539+0000","last_fullsized":"2024-03-23T15:47:28.686539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679986+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:47:28.679986+0000","last_peered":"2024-03-23T15:47:28.679986+0000","last_clean":"2024-03-23T15:47:28.679986+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:47:28.679986+0000","last_undegraded":"2024-03-23T15:47:28.679986+0000","last_fullsized":"2024-03-23T15:47:28.679986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465648+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:47:28.465648+0000","last_peered":"2024-03-23T15:47:28.465648+0000","last_clean":"2024-03-23T15:47:28.465648+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:47:28.465648+0000","last_undegraded":"2024-03-23T15:47:28.465648+0000","last_fullsized":"2024-03-23T15:47:28.465648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410906+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:47:28.410906+0000","last_peered":"2024-03-23T15:47:28.410906+0000","last_clean":"2024-03-23T15:47:28.410906+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:47:28.410906+0000","last_undegraded":"2024-03-23T15:47:28.410906+0000","last_fullsized":"2024-03-23T15:47:28.410906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416609+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:47:28.416609+0000","last_peered":"2024-03-23T15:47:28.416609+0000","last_clean":"2024-03-23T15:47:28.416609+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:47:28.416609+0000","last_undegraded":"2024-03-23T15:47:28.416609+0000","last_fullsized":"2024-03-23T15:47:28.416609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411310+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:47:28.411310+0000","last_peered":"2024-03-23T15:47:28.411310+0000","last_clean":"2024-03-23T15:47:28.411310+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:47:28.411310+0000","last_undegraded":"2024-03-23T15:47:28.411310+0000","last_fullsized":"2024-03-23T15:47:28.411310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682742+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:47:28.682742+0000","last_peered":"2024-03-23T15:47:28.682742+0000","last_clean":"2024-03-23T15:47:28.682742+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:47:28.682742+0000","last_undegraded":"2024-03-23T15:47:28.682742+0000","last_fullsized":"2024-03-23T15:47:28.682742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747920+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:47:28.747920+0000","last_peered":"2024-03-23T15:47:28.747920+0000","last_clean":"2024-03-23T15:47:28.747920+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:47:28.747920+0000","last_undegraded":"2024-03-23T15:47:28.747920+0000","last_fullsized":"2024-03-23T15:47:28.747920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747807+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:47:28.747807+0000","last_peered":"2024-03-23T15:47:28.747807+0000","last_clean":"2024-03-23T15:47:28.747807+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:47:28.747807+0000","last_undegraded":"2024-03-23T15:47:28.747807+0000","last_fullsized":"2024-03-23T15:47:28.747807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411018+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:47:28.411018+0000","last_peered":"2024-03-23T15:47:28.411018+0000","last_clean":"2024-03-23T15:47:28.411018+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:47:28.411018+0000","last_undegraded":"2024-03-23T15:47:28.411018+0000","last_fullsized":"2024-03-23T15:47:28.411018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416777+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:47:28.416777+0000","last_peered":"2024-03-23T15:47:28.416777+0000","last_clean":"2024-03-23T15:47:28.416777+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:47:28.416777+0000","last_undegraded":"2024-03-23T15:47:28.416777+0000","last_fullsized":"2024-03-23T15:47:28.416777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681919+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:47:28.681919+0000","last_peered":"2024-03-23T15:47:28.681919+0000","last_clean":"2024-03-23T15:47:28.681919+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:47:28.681919+0000","last_undegraded":"2024-03-23T15:47:28.681919+0000","last_fullsized":"2024-03-23T15:47:28.681919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682193+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:47:28.682193+0000","last_peered":"2024-03-23T15:47:28.682193+0000","last_clean":"2024-03-23T15:47:28.682193+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:47:28.682193+0000","last_undegraded":"2024-03-23T15:47:28.682193+0000","last_fullsized":"2024-03-23T15:47:28.682193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465791+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:47:28.465791+0000","last_peered":"2024-03-23T15:47:28.465791+0000","last_clean":"2024-03-23T15:47:28.465791+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:47:28.465791+0000","last_undegraded":"2024-03-23T15:47:28.465791+0000","last_fullsized":"2024-03-23T15:47:28.465791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.582464+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:47:33.582464+0000","last_peered":"2024-03-23T15:47:33.582464+0000","last_clean":"2024-03-23T15:47:33.582464+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:47:33.582464+0000","last_undegraded":"2024-03-23T15:47:33.582464+0000","last_fullsized":"2024-03-23T15:47:33.582464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680247+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:47:28.680247+0000","last_peered":"2024-03-23T15:47:28.680247+0000","last_clean":"2024-03-23T15:47:28.680247+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:47:28.680247+0000","last_undegraded":"2024-03-23T15:47:28.680247+0000","last_fullsized":"2024-03-23T15:47:28.680247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.702619+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:47:35.702619+0000","last_peered":"2024-03-23T15:47:35.702619+0000","last_clean":"2024-03-23T15:47:35.702619+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:47:35.702619+0000","last_undegraded":"2024-03-23T15:47:35.702619+0000","last_fullsized":"2024-03-23T15:47:35.702619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.694180+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:47:35.694180+0000","last_peered":"2024-03-23T15:47:35.694180+0000","last_clean":"2024-03-23T15:47:35.694180+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:47:35.694180+0000","last_undegraded":"2024-03-23T15:47:35.694180+0000","last_fullsized":"2024-03-23T15:47:35.694180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411290+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:47:28.411290+0000","last_peered":"2024-03-23T15:47:28.411290+0000","last_clean":"2024-03-23T15:47:28.411290+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:47:28.411290+0000","last_undegraded":"2024-03-23T15:47:28.411290+0000","last_fullsized":"2024-03-23T15:47:28.411290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416755+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:47:28.416755+0000","last_peered":"2024-03-23T15:47:28.416755+0000","last_clean":"2024-03-23T15:47:28.416755+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:47:28.416755+0000","last_undegraded":"2024-03-23T15:47:28.416755+0000","last_fullsized":"2024-03-23T15:47:28.416755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679735+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:47:28.679735+0000","last_peered":"2024-03-23T15:47:28.679735+0000","last_clean":"2024-03-23T15:47:28.679735+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:47:28.679735+0000","last_undegraded":"2024-03-23T15:47:28.679735+0000","last_fullsized":"2024-03-23T15:47:28.679735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.590599+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:47:33.590599+0000","last_peered":"2024-03-23T15:47:33.590599+0000","last_clean":"2024-03-23T15:47:33.590599+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:47:33.590599+0000","last_undegraded":"2024-03-23T15:47:33.590599+0000","last_fullsized":"2024-03-23T15:47:33.590599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682790+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:47:28.682790+0000","last_peered":"2024-03-23T15:47:28.682790+0000","last_clean":"2024-03-23T15:47:28.682790+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:47:28.682790+0000","last_undegraded":"2024-03-23T15:47:28.682790+0000","last_fullsized":"2024-03-23T15:47:28.682790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.600073+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:47:33.600073+0000","last_peered":"2024-03-23T15:47:33.600073+0000","last_clean":"2024-03-23T15:47:33.600073+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:47:33.600073+0000","last_undegraded":"2024-03-23T15:47:33.600073+0000","last_fullsized":"2024-03-23T15:47:33.600073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411427+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:47:28.411427+0000","last_peered":"2024-03-23T15:47:28.411427+0000","last_clean":"2024-03-23T15:47:28.411427+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:47:28.411427+0000","last_undegraded":"2024-03-23T15:47:28.411427+0000","last_fullsized":"2024-03-23T15:47:28.411427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.685208+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:47:28.685208+0000","last_peered":"2024-03-23T15:47:28.685208+0000","last_clean":"2024-03-23T15:47:28.685208+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:47:28.685208+0000","last_undegraded":"2024-03-23T15:47:28.685208+0000","last_fullsized":"2024-03-23T15:47:28.685208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416762+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:47:28.416762+0000","last_peered":"2024-03-23T15:47:28.416762+0000","last_clean":"2024-03-23T15:47:28.416762+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:47:28.416762+0000","last_undegraded":"2024-03-23T15:47:28.416762+0000","last_fullsized":"2024-03-23T15:47:28.416762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.501510+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:47:33.501510+0000","last_peered":"2024-03-23T15:47:33.501510+0000","last_clean":"2024-03-23T15:47:33.501510+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:47:33.501510+0000","last_undegraded":"2024-03-23T15:47:33.501510+0000","last_fullsized":"2024-03-23T15:47:33.501510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686519+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:47:28.686519+0000","last_peered":"2024-03-23T15:47:28.686519+0000","last_clean":"2024-03-23T15:47:28.686519+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:47:28.686519+0000","last_undegraded":"2024-03-23T15:47:28.686519+0000","last_fullsized":"2024-03-23T15:47:28.686519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.682429+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:47:35.682429+0000","last_peered":"2024-03-23T15:47:35.682429+0000","last_clean":"2024-03-23T15:47:35.682429+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:47:35.682429+0000","last_undegraded":"2024-03-23T15:47:35.682429+0000","last_fullsized":"2024-03-23T15:47:35.682429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747624+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:47:28.747624+0000","last_peered":"2024-03-23T15:47:28.747624+0000","last_clean":"2024-03-23T15:47:28.747624+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:47:28.747624+0000","last_undegraded":"2024-03-23T15:47:28.747624+0000","last_fullsized":"2024-03-23T15:47:28.747624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.498966+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:47:33.498966+0000","last_peered":"2024-03-23T15:47:33.498966+0000","last_clean":"2024-03-23T15:47:33.498966+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:47:33.498966+0000","last_undegraded":"2024-03-23T15:47:33.498966+0000","last_fullsized":"2024-03-23T15:47:33.498966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416928+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:47:28.416928+0000","last_peered":"2024-03-23T15:47:28.416928+0000","last_clean":"2024-03-23T15:47:28.416928+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:47:28.416928+0000","last_undegraded":"2024-03-23T15:47:28.416928+0000","last_fullsized":"2024-03-23T15:47:28.416928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.559806+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:47:35.559806+0000","last_peered":"2024-03-23T15:47:35.559806+0000","last_clean":"2024-03-23T15:47:35.559806+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:47:35.559806+0000","last_undegraded":"2024-03-23T15:47:35.559806+0000","last_fullsized":"2024-03-23T15:47:35.559806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465425+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:47:28.465425+0000","last_peered":"2024-03-23T15:47:28.465425+0000","last_clean":"2024-03-23T15:47:28.465425+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:47:28.465425+0000","last_undegraded":"2024-03-23T15:47:28.465425+0000","last_fullsized":"2024-03-23T15:47:28.465425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.569070+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:47:35.569070+0000","last_peered":"2024-03-23T15:47:35.569070+0000","last_clean":"2024-03-23T15:47:35.569070+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:47:35.569070+0000","last_undegraded":"2024-03-23T15:47:35.569070+0000","last_fullsized":"2024-03-23T15:47:35.569070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416623+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:47:28.416623+0000","last_peered":"2024-03-23T15:47:28.416623+0000","last_clean":"2024-03-23T15:47:28.416623+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:47:28.416623+0000","last_undegraded":"2024-03-23T15:47:28.416623+0000","last_fullsized":"2024-03-23T15:47:28.416623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686786+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:47:28.686786+0000","last_peered":"2024-03-23T15:47:28.686786+0000","last_clean":"2024-03-23T15:47:28.686786+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:47:28.686786+0000","last_undegraded":"2024-03-23T15:47:28.686786+0000","last_fullsized":"2024-03-23T15:47:28.686786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682875+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:47:28.682875+0000","last_peered":"2024-03-23T15:47:28.682875+0000","last_clean":"2024-03-23T15:47:28.682875+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:47:28.682875+0000","last_undegraded":"2024-03-23T15:47:28.682875+0000","last_fullsized":"2024-03-23T15:47:28.682875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.683735+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:47:28.683735+0000","last_peered":"2024-03-23T15:47:28.683735+0000","last_clean":"2024-03-23T15:47:28.683735+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:47:28.683735+0000","last_undegraded":"2024-03-23T15:47:28.683735+0000","last_fullsized":"2024-03-23T15:47:28.683735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686676+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:47:28.686676+0000","last_peered":"2024-03-23T15:47:28.686676+0000","last_clean":"2024-03-23T15:47:28.686676+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:47:28.686676+0000","last_undegraded":"2024-03-23T15:47:28.686676+0000","last_fullsized":"2024-03-23T15:47:28.686676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411312+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:47:28.411312+0000","last_peered":"2024-03-23T15:47:28.411312+0000","last_clean":"2024-03-23T15:47:28.411312+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:47:28.411312+0000","last_undegraded":"2024-03-23T15:47:28.411312+0000","last_fullsized":"2024-03-23T15:47:28.411312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686633+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:47:28.686633+0000","last_peered":"2024-03-23T15:47:28.686633+0000","last_clean":"2024-03-23T15:47:28.686633+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:47:28.686633+0000","last_undegraded":"2024-03-23T15:47:28.686633+0000","last_fullsized":"2024-03-23T15:47:28.686633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.699873+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:47:35.699873+0000","last_peered":"2024-03-23T15:47:35.699873+0000","last_clean":"2024-03-23T15:47:35.699873+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:47:35.699873+0000","last_undegraded":"2024-03-23T15:47:35.699873+0000","last_fullsized":"2024-03-23T15:47:35.699873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417319+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:47:28.417319+0000","last_peered":"2024-03-23T15:47:28.417319+0000","last_clean":"2024-03-23T15:47:28.417319+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:47:28.417319+0000","last_undegraded":"2024-03-23T15:47:28.417319+0000","last_fullsized":"2024-03-23T15:47:28.417319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":171,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465446+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:47:28.465446+0000","last_peered":"2024-03-23T15:47:28.465446+0000","last_clean":"2024-03-23T15:47:28.465446+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:47:28.465446+0000","last_undegraded":"2024-03-23T15:47:28.465446+0000","last_fullsized":"2024-03-23T15:47:28.465446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.499108+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:47:33.499108+0000","last_peered":"2024-03-23T15:47:33.499108+0000","last_clean":"2024-03-23T15:47:33.499108+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:47:33.499108+0000","last_undegraded":"2024-03-23T15:47:33.499108+0000","last_fullsized":"2024-03-23T15:47:33.499108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679960+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:47:28.679960+0000","last_peered":"2024-03-23T15:47:28.679960+0000","last_clean":"2024-03-23T15:47:28.679960+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:47:28.679960+0000","last_undegraded":"2024-03-23T15:47:28.679960+0000","last_fullsized":"2024-03-23T15:47:28.679960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.699460+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:47:35.699460+0000","last_peered":"2024-03-23T15:47:35.699460+0000","last_clean":"2024-03-23T15:47:35.699460+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:47:35.699460+0000","last_undegraded":"2024-03-23T15:47:35.699460+0000","last_fullsized":"2024-03-23T15:47:35.699460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747588+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:47:28.747588+0000","last_peered":"2024-03-23T15:47:28.747588+0000","last_clean":"2024-03-23T15:47:28.747588+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:47:28.747588+0000","last_undegraded":"2024-03-23T15:47:28.747588+0000","last_fullsized":"2024-03-23T15:47:28.747588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.688236+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:47:35.688236+0000","last_peered":"2024-03-23T15:47:35.688236+0000","last_clean":"2024-03-23T15:47:35.688236+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:47:35.688236+0000","last_undegraded":"2024-03-23T15:47:35.688236+0000","last_fullsized":"2024-03-23T15:47:35.688236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411099+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:47:28.411099+0000","last_peered":"2024-03-23T15:47:28.411099+0000","last_clean":"2024-03-23T15:47:28.411099+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:47:28.411099+0000","last_undegraded":"2024-03-23T15:47:28.411099+0000","last_fullsized":"2024-03-23T15:47:28.411099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417103+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:47:28.417103+0000","last_peered":"2024-03-23T15:47:28.417103+0000","last_clean":"2024-03-23T15:47:28.417103+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:47:28.417103+0000","last_undegraded":"2024-03-23T15:47:28.417103+0000","last_fullsized":"2024-03-23T15:47:28.417103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686285+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:47:28.686285+0000","last_peered":"2024-03-23T15:47:28.686285+0000","last_clean":"2024-03-23T15:47:28.686285+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:47:28.686285+0000","last_undegraded":"2024-03-23T15:47:28.686285+0000","last_fullsized":"2024-03-23T15:47:28.686285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411034+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:47:28.411034+0000","last_peered":"2024-03-23T15:47:28.411034+0000","last_clean":"2024-03-23T15:47:28.411034+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:47:28.411034+0000","last_undegraded":"2024-03-23T15:47:28.411034+0000","last_fullsized":"2024-03-23T15:47:28.411034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682390+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:47:28.682390+0000","last_peered":"2024-03-23T15:47:28.682390+0000","last_clean":"2024-03-23T15:47:28.682390+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:47:28.682390+0000","last_undegraded":"2024-03-23T15:47:28.682390+0000","last_fullsized":"2024-03-23T15:47:28.682390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417435+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:47:28.417435+0000","last_peered":"2024-03-23T15:47:28.417435+0000","last_clean":"2024-03-23T15:47:28.417435+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:47:28.417435+0000","last_undegraded":"2024-03-23T15:47:28.417435+0000","last_fullsized":"2024-03-23T15:47:28.417435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747808+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:47:28.747808+0000","last_peered":"2024-03-23T15:47:28.747808+0000","last_clean":"2024-03-23T15:47:28.747808+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:47:28.747808+0000","last_undegraded":"2024-03-23T15:47:28.747808+0000","last_fullsized":"2024-03-23T15:47:28.747808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.562394+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:47:35.562394+0000","last_peered":"2024-03-23T15:47:35.562394+0000","last_clean":"2024-03-23T15:47:35.562394+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:47:35.562394+0000","last_undegraded":"2024-03-23T15:47:35.562394+0000","last_fullsized":"2024-03-23T15:47:35.562394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680240+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:47:28.680240+0000","last_peered":"2024-03-23T15:47:28.680240+0000","last_clean":"2024-03-23T15:47:28.680240+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:47:28.680240+0000","last_undegraded":"2024-03-23T15:47:28.680240+0000","last_fullsized":"2024-03-23T15:47:28.680240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.496287+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:47:33.496287+0000","last_peered":"2024-03-23T15:47:33.496287+0000","last_clean":"2024-03-23T15:47:33.496287+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:47:33.496287+0000","last_undegraded":"2024-03-23T15:47:33.496287+0000","last_fullsized":"2024-03-23T15:47:33.496287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680148+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:47:28.680148+0000","last_peered":"2024-03-23T15:47:28.680148+0000","last_clean":"2024-03-23T15:47:28.680148+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:47:28.680148+0000","last_undegraded":"2024-03-23T15:47:28.680148+0000","last_fullsized":"2024-03-23T15:47:28.680148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.602195+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:47:33.602195+0000","last_peered":"2024-03-23T15:47:33.602195+0000","last_clean":"2024-03-23T15:47:33.602195+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:47:33.602195+0000","last_undegraded":"2024-03-23T15:47:33.602195+0000","last_fullsized":"2024-03-23T15:47:33.602195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416764+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:47:28.416764+0000","last_peered":"2024-03-23T15:47:28.416764+0000","last_clean":"2024-03-23T15:47:28.416764+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:47:28.416764+0000","last_undegraded":"2024-03-23T15:47:28.416764+0000","last_fullsized":"2024-03-23T15:47:28.416764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.501475+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:47:33.501475+0000","last_peered":"2024-03-23T15:47:33.501475+0000","last_clean":"2024-03-23T15:47:33.501475+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:47:33.501475+0000","last_undegraded":"2024-03-23T15:47:33.501475+0000","last_fullsized":"2024-03-23T15:47:33.501475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679874+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:47:28.679874+0000","last_peered":"2024-03-23T15:47:28.679874+0000","last_clean":"2024-03-23T15:47:28.679874+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:47:28.679874+0000","last_undegraded":"2024-03-23T15:47:28.679874+0000","last_fullsized":"2024-03-23T15:47:28.679874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747779+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:47:28.747779+0000","last_peered":"2024-03-23T15:47:28.747779+0000","last_clean":"2024-03-23T15:47:28.747779+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:47:28.747779+0000","last_undegraded":"2024-03-23T15:47:28.747779+0000","last_fullsized":"2024-03-23T15:47:28.747779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682112+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:47:28.682112+0000","last_peered":"2024-03-23T15:47:28.682112+0000","last_clean":"2024-03-23T15:47:28.682112+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:47:28.682112+0000","last_undegraded":"2024-03-23T15:47:28.682112+0000","last_fullsized":"2024-03-23T15:47:28.682112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.493555+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:47:33.493555+0000","last_peered":"2024-03-23T15:47:33.493555+0000","last_clean":"2024-03-23T15:47:33.493555+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:47:33.493555+0000","last_undegraded":"2024-03-23T15:47:33.493555+0000","last_fullsized":"2024-03-23T15:47:33.493555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416913+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:47:28.416913+0000","last_peered":"2024-03-23T15:47:28.416913+0000","last_clean":"2024-03-23T15:47:28.416913+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:47:28.416913+0000","last_undegraded":"2024-03-23T15:47:28.416913+0000","last_fullsized":"2024-03-23T15:47:28.416913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416753+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:47:28.416753+0000","last_peered":"2024-03-23T15:47:28.416753+0000","last_clean":"2024-03-23T15:47:28.416753+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:47:28.416753+0000","last_undegraded":"2024-03-23T15:47:28.416753+0000","last_fullsized":"2024-03-23T15:47:28.416753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686726+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:47:28.686726+0000","last_peered":"2024-03-23T15:47:28.686726+0000","last_clean":"2024-03-23T15:47:28.686726+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:47:28.686726+0000","last_undegraded":"2024-03-23T15:47:28.686726+0000","last_fullsized":"2024-03-23T15:47:28.686726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465591+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:47:28.465591+0000","last_peered":"2024-03-23T15:47:28.465591+0000","last_clean":"2024-03-23T15:47:28.465591+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:47:28.465591+0000","last_undegraded":"2024-03-23T15:47:28.465591+0000","last_fullsized":"2024-03-23T15:47:28.465591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.559483+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:47:35.559483+0000","last_peered":"2024-03-23T15:47:35.559483+0000","last_clean":"2024-03-23T15:47:35.559483+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:47:35.559483+0000","last_undegraded":"2024-03-23T15:47:35.559483+0000","last_fullsized":"2024-03-23T15:47:35.559483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411323+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:47:28.411323+0000","last_peered":"2024-03-23T15:47:28.411323+0000","last_clean":"2024-03-23T15:47:28.411323+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:47:28.411323+0000","last_undegraded":"2024-03-23T15:47:28.411323+0000","last_fullsized":"2024-03-23T15:47:28.411323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747051+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:47:28.747051+0000","last_peered":"2024-03-23T15:47:28.747051+0000","last_clean":"2024-03-23T15:47:28.747051+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:47:28.747051+0000","last_undegraded":"2024-03-23T15:47:28.747051+0000","last_fullsized":"2024-03-23T15:47:28.747051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411212+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:47:28.411212+0000","last_peered":"2024-03-23T15:47:28.411212+0000","last_clean":"2024-03-23T15:47:28.411212+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:47:28.411212+0000","last_undegraded":"2024-03-23T15:47:28.411212+0000","last_fullsized":"2024-03-23T15:47:28.411212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.567079+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:47:35.567079+0000","last_peered":"2024-03-23T15:47:35.567079+0000","last_clean":"2024-03-23T15:47:35.567079+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:47:35.567079+0000","last_undegraded":"2024-03-23T15:47:35.567079+0000","last_fullsized":"2024-03-23T15:47:35.567079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":10444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":303104,"data_stored":667163,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":262144,"data_stored":79171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":666519,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T15:47:37.112 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T15:47:37.112 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-23T15:47:37.112 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-23T15:47:37.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-23T15:47:37.113 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-23T15:47:37.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.249+0000 7fa13f6d4640 1 Processor -- start 2024-03-23T15:47:37.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.250+0000 7fa13f6d4640 1 -- start start 2024-03-23T15:47:37.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.250+0000 7fa13f6d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1380c6000 0x7fa1380c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:37.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.250+0000 7fa13f6d4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1380cbcb0 con 0x7fa13815a2e0 2024-03-23T15:47:37.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.250+0000 7fa13f6d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13815b280 con 0x7fa1381544b0 2024-03-23T15:47:37.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.250+0000 7fa13f6d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa1380c5be0 con 0x7fa1380c6000 2024-03-23T15:47:37.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.251+0000 7fa13cc6f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa13815a2e0 0x7fa13806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36510/0 (socket says 172.21.15.129:36510) 2024-03-23T15:47:37.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.251+0000 7fa13cc6f640 1 -- 172.21.15.129:0/2188523292 learned_addr learned my addr 172.21.15.129:0/2188523292 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:37.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.251+0000 7fa13dc71640 1 --2- 172.21.15.129:0/2188523292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1380c6000 0x7fa1380c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.253+0000 7fa13dc71640 1 -- 172.21.15.129:0/2188523292 >> v1:172.21.15.157:6789/0 conn(0x7fa1381544b0 legacy=0x7fa1380caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.253+0000 7fa13dc71640 1 -- 172.21.15.129:0/2188523292 >> v1:172.21.15.129:6789/0 conn(0x7fa13815a2e0 legacy=0x7fa13806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.253+0000 7fa13dc71640 1 -- 172.21.15.129:0/2188523292 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13815a9e0 con 0x7fa1380c6000 2024-03-23T15:47:37.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.255+0000 7fa13dc71640 1 --2- 172.21.15.129:0/2188523292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1380c6000 0x7fa1380c6470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa130009dc0 tx=0x7fa13002bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b80a20da61e1b0a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:37.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.259+0000 7fa12ffff640 1 -- 172.21.15.129:0/2188523292 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13002cd50 con 0x7fa1380c6000 2024-03-23T15:47:37.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.259+0000 7fa12ffff640 1 -- 172.21.15.129:0/2188523292 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa130034440 con 0x7fa1380c6000 2024-03-23T15:47:37.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.259+0000 7fa12ffff640 1 -- 172.21.15.129:0/2188523292 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300347d0 con 0x7fa1380c6000 2024-03-23T15:47:37.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 -- 172.21.15.129:0/2188523292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1380c6000 msgr2=0x7fa1380c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/2188523292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1380c6000 0x7fa1380c6470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa130009dc0 tx=0x7fa13002bfa0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 -- 172.21.15.129:0/2188523292 shutdown_connections 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/2188523292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1380c6000 0x7fa1380c6470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 -- 172.21.15.129:0/2188523292 >> 172.21.15.129:0/2188523292 conn(0x7fa138061b50 msgr2=0x7fa138061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 -- 172.21.15.129:0/2188523292 shutdown_connections 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.260+0000 7fa13f6d4640 1 -- 172.21.15.129:0/2188523292 wait complete. 2024-03-23T15:47:37.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.261+0000 7fa13f6d4640 1 Processor -- start 2024-03-23T15:47:37.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.261+0000 7fa13f6d4640 1 -- start start 2024-03-23T15:47:37.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13f6d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1380c6000 0x7fa13806bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:37.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13f6d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1381544b0 0x7fa1380cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:37.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13f6d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 0x7fa1380cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:37.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13f6d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1380cda40 con 0x7fa1381544b0 2024-03-23T15:47:37.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13f6d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1380cdd20 con 0x7fa1380c6000 2024-03-23T15:47:37.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13f6d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa1380d02a0 con 0x7fa13815a2e0 2024-03-23T15:47:37.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13dc71640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 0x7fa1380cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:37.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13cc6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1381544b0 0x7fa1380cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:37.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13d470640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1380c6000 0x7fa13806bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:37.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13dc71640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 0x7fa1380cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35618/0 (socket says 172.21.15.129:35618) 2024-03-23T15:47:37.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.262+0000 7fa13dc71640 1 -- 172.21.15.129:0/1487614395 learned_addr learned my addr 172.21.15.129:0/1487614395 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.266+0000 7fa13cc6f640 1 -- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 msgr2=0x7fa1380cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.266+0000 7fa13cc6f640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 0x7fa1380cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.266+0000 7fa13cc6f640 1 -- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1380c6000 msgr2=0x7fa13806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.266+0000 7fa13cc6f640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1380c6000 0x7fa13806bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.266+0000 7fa13cc6f640 1 -- 172.21.15.129:0/1487614395 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa130009a70 con 0x7fa1381544b0 2024-03-23T15:47:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.266+0000 7fa13dc71640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 0x7fa1380cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:47:37.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.268+0000 7fa13cc6f640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1381544b0 0x7fa1380cf570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c4a0 tx=0x7fa12800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:37.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.272+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128005ce0 con 0x7fa1381544b0 2024-03-23T15:47:37.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.272+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa128005e40 con 0x7fa1381544b0 2024-03-23T15:47:37.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.272+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1381f44e0 con 0x7fa1381544b0 2024-03-23T15:47:37.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.272+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1280072d0 con 0x7fa1381544b0 2024-03-23T15:47:37.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.272+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1380d07f0 con 0x7fa1381544b0 2024-03-23T15:47:37.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.273+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa138075740 con 0x7fa1381544b0 2024-03-23T15:47:37.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.303+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa128007430 con 0x7fa1381544b0 2024-03-23T15:47:37.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.303+0000 7fa12dffb640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa11c06b860 0x7fa11c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:37.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.304+0000 7fa13d470640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa11c06b860 0x7fa11c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:37.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.304+0000 7fa13d470640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa11c06b860 0x7fa11c06dd20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa134009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:37.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.308+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa128058350 con 0x7fa1381544b0 2024-03-23T15:47:37.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.558+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa1280545f0 con 0x7fa1381544b0 2024-03-23T15:47:37.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.818+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa138151ab0 con 0x7fa1381544b0 2024-03-23T15:47:37.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.987+0000 7fa12dffb640 1 -- 172.21.15.129:0/1487614395 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+171790 (secure 0 0 0) 0x7fa128017020 con 0x7fa1381544b0 2024-03-23T15:47:37.992 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa11c06b860 msgr2=0x7fa11c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa11c06b860 0x7fa11c06dd20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa134009040 comp rx=0 tx=0).stop 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1381544b0 msgr2=0x7fa1380cf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1381544b0 0x7fa1380cf570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c4a0 tx=0x7fa12800c960 comp rx=0 tx=0).stop 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 shutdown_connections 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa13815a2e0 0x7fa1380cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1381544b0 0x7fa1380cf570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa11c06b860 0x7fa11c06dd20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 --2- 172.21.15.129:0/1487614395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1380c6000 0x7fa13806bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 >> 172.21.15.129:0/1487614395 conn(0x7fa138061b50 msgr2=0x7fa13814a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 shutdown_connections 2024-03-23T15:47:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:37.990+0000 7fa13f6d4640 1 -- 172.21.15.129:0/1487614395 wait complete. 2024-03-23T15:47:38.005 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":2961890009},{"type":"v1","addr":"172.21.15.157:6801","nonce":2961890009}]},"active_addr":"172.21.15.157:6801/2961890009","active_change":"2024-03-23T15:47:02.110378+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":2269362449}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":683530143}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":3899155596}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":287904277}]}]} 2024-03-23T15:47:38.008 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-23T15:47:38.008 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-23T15:47:38.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:38.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.115+0000 7fb71fed0640 1 Processor -- start 2024-03-23T15:47:38.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.115+0000 7fb71fed0640 1 -- start start 2024-03-23T15:47:38.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.116+0000 7fb71fed0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb718158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.116+0000 7fb71fed0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb718071530 0x7fb7180c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.116+0000 7fb71fed0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7181591e0 con 0x7fb718141630 2024-03-23T15:47:38.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.116+0000 7fb71fed0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb71815c740 con 0x7fb718071530 2024-03-23T15:47:38.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.116+0000 7fb71fed0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb718141a10 con 0x7fb7180c8f70 2024-03-23T15:47:38.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.117+0000 7fb71d46b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb718158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.117+0000 7fb71e46d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb718071530 0x7fb7180c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.117+0000 7fb71d46b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb718158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35632/0 (socket says 172.21.15.129:35632) 2024-03-23T15:47:38.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.117+0000 7fb71d46b640 1 -- 172.21.15.129:0/1766604777 learned_addr learned my addr 172.21.15.129:0/1766604777 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:38.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.119+0000 7fb71d46b640 1 -- 172.21.15.129:0/1766604777 >> v2:172.21.15.129:3300/0 conn(0x7fb718071530 msgr2=0x7fb7180c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:38.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.119+0000 7fb71d46b640 1 --2- 172.21.15.129:0/1766604777 >> v2:172.21.15.129:3300/0 conn(0x7fb718071530 0x7fb7180c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.119+0000 7fb71d46b640 1 -- 172.21.15.129:0/1766604777 >> v1:172.21.15.129:6789/0 conn(0x7fb718141630 legacy=0x7fb71815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.120+0000 7fb71d46b640 1 -- 172.21.15.129:0/1766604777 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71815bba0 con 0x7fb7180c8f70 2024-03-23T15:47:38.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.121+0000 7fb71d46b640 1 --2- 172.21.15.129:0/1766604777 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb718158ca0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb708009870 tx=0x7fb70802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=311536fd8b46536c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:38.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.125+0000 7fb71cc6a640 1 -- 172.21.15.129:0/1766604777 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70802f840 con 0x7fb7180c8f70 2024-03-23T15:47:38.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.125+0000 7fb71cc6a640 1 -- 172.21.15.129:0/1766604777 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70802f9a0 con 0x7fb7180c8f70 2024-03-23T15:47:38.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.125+0000 7fb71cc6a640 1 -- 172.21.15.129:0/1766604777 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70802fd30 con 0x7fb7180c8f70 2024-03-23T15:47:38.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.126+0000 7fb71fed0640 1 -- 172.21.15.129:0/1766604777 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 msgr2=0x7fb718158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.126+0000 7fb71fed0640 1 --2- 172.21.15.129:0/1766604777 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb718158ca0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb708009870 tx=0x7fb70802ba20 comp rx=0 tx=0).stop 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.126+0000 7fb71fed0640 1 -- 172.21.15.129:0/1766604777 shutdown_connections 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.126+0000 7fb71fed0640 1 --2- 172.21.15.129:0/1766604777 >> v2:172.21.15.129:3300/0 conn(0x7fb718071530 0x7fb7180c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.126+0000 7fb71fed0640 1 --2- 172.21.15.129:0/1766604777 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb718158ca0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.126+0000 7fb71fed0640 1 -- 172.21.15.129:0/1766604777 >> 172.21.15.129:0/1766604777 conn(0x7fb7180c8370 msgr2=0x7fb7180c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.127+0000 7fb71fed0640 1 -- 172.21.15.129:0/1766604777 shutdown_connections 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.127+0000 7fb71fed0640 1 -- 172.21.15.129:0/1766604777 wait complete. 2024-03-23T15:47:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.128+0000 7fb71fed0640 1 Processor -- start 2024-03-23T15:47:38.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.128+0000 7fb71fed0640 1 -- start start 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71fed0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb718071530 0x7fb71813f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71fed0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb7181ee540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71fed0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb718141630 0x7fb7181eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71fed0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7181558a0 con 0x7fb718141630 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71fed0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb718155df0 con 0x7fb718071530 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71fed0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7181f1ea0 con 0x7fb7180c8f70 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71e46d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb718141630 0x7fb7181eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71d46b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb7181ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71dc6c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb718071530 0x7fb71813f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71d46b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb7181ee540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35648/0 (socket says 172.21.15.129:35648) 2024-03-23T15:47:38.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.129+0000 7fb71d46b640 1 -- 172.21.15.129:0/820961507 learned_addr learned my addr 172.21.15.129:0/820961507 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:38.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.133+0000 7fb71e46d640 1 -- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 msgr2=0x7fb7181ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.133+0000 7fb71e46d640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb7181ee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.133+0000 7fb71e46d640 1 -- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb718071530 msgr2=0x7fb71813f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.133+0000 7fb71e46d640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb718071530 0x7fb71813f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.133+0000 7fb71e46d640 1 -- 172.21.15.129:0/820961507 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7080094d0 con 0x7fb718141630 2024-03-23T15:47:38.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.135+0000 7fb71e46d640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb718141630 0x7fb7181eea80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb71000efc0 tx=0x7fb71000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:38.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.139+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb710009c00 con 0x7fb718141630 2024-03-23T15:47:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.139+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb71000f040 con 0x7fb718141630 2024-03-23T15:47:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.139+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7100146c0 con 0x7fb718141630 2024-03-23T15:47:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.139+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7181f2180 con 0x7fb718141630 2024-03-23T15:47:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.139+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7181ee020 con 0x7fb718141630 2024-03-23T15:47:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.140+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7181ede80 con 0x7fb718141630 2024-03-23T15:47:38.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.160+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb710014820 con 0x7fb718141630 2024-03-23T15:47:38.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.161+0000 7fb70e7fc640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb6fc06b860 0x7fb6fc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.161+0000 7fb71dc6c640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb6fc06b860 0x7fb6fc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.162+0000 7fb71dc6c640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb6fc06b860 0x7fb6fc06dd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb7140016e0 tx=0x7fb714009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:38.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.409+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb710059b80 con 0x7fb718141630 2024-03-23T15:47:38.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.409+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb710055e20 con 0x7fb718141630 2024-03-23T15:47:38.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.618+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7181ede80 con 0x7fb718141630 2024-03-23T15:47:38.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.688+0000 7fb70e7fc640 1 -- 172.21.15.129:0/820961507 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fb7181ede80 con 0x7fb718141630 2024-03-23T15:47:38.692 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:38.692 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:38.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb6fc06b860 msgr2=0x7fb6fc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb6fc06b860 0x7fb6fc06dd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb7140016e0 tx=0x7fb714009040 comp rx=0 tx=0).stop 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb718141630 msgr2=0x7fb7181eea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb718141630 0x7fb7181eea80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb71000efc0 tx=0x7fb71000c490 comp rx=0 tx=0).stop 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 shutdown_connections 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb718141630 0x7fb7181eea80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7180c8f70 0x7fb7181ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb6fc06b860 0x7fb6fc06dd20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 --2- 172.21.15.129:0/820961507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb718071530 0x7fb71813f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 >> 172.21.15.129:0/820961507 conn(0x7fb7180c8370 msgr2=0x7fb71813d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 shutdown_connections 2024-03-23T15:47:38.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.691+0000 7fb71fed0640 1 -- 172.21.15.129:0/820961507 wait complete. 2024-03-23T15:47:38.710 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-23T15:47:38.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:38.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.817+0000 7f4f022b5640 1 Processor -- start 2024-03-23T15:47:38.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.817+0000 7f4f022b5640 1 -- start start 2024-03-23T15:47:38.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.818+0000 7f4f022b5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4efc0c7380 0x7f4efc0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.818+0000 7f4f022b5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4efc0c3bd0 con 0x7f4efc156f60 2024-03-23T15:47:38.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.818+0000 7f4f022b5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4efc0c3740 con 0x7f4efc0c8e90 2024-03-23T15:47:38.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.818+0000 7f4f022b5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4efc0c7ca0 con 0x7f4efc0c7380 2024-03-23T15:47:38.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.864+0000 7f4f00852640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4efc156f60 0x7f4efc157370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36528/0 (socket says 172.21.15.129:36528) 2024-03-23T15:47:38.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.864+0000 7f4f00852640 1 -- 172.21.15.129:0/1769066086 learned_addr learned my addr 172.21.15.129:0/1769066086 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.867+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3137656601 0 0) 0x7f4efc0c3bd0 con 0x7f4efc156f60 2024-03-23T15:47:38.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.867+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee4003520 con 0x7f4efc156f60 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1599126530 0 0) 0x7f4ee4003520 con 0x7f4efc156f60 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 >> v2:172.21.15.157:3300/0 conn(0x7f4efc0c7380 msgr2=0x7f4efc0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 --2- 172.21.15.129:0/1769066086 >> v2:172.21.15.157:3300/0 conn(0x7f4efc0c7380 0x7f4efc0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 >> v1:172.21.15.157:6789/0 conn(0x7f4efc0c8e90 legacy=0x7f4efc0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4efc0c80c0 con 0x7f4efc156f60 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f4ef40019f0 con 0x7f4efc156f60 2024-03-23T15:47:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 >> v1:172.21.15.129:6789/0 conn(0x7f4efc156f60 legacy=0x7f4efc157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee40039c0 0x7f4ee4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee40047c0 0x7f4ee4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40052a0 0x7f4ee400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4efc0c7ca0 con 0x7f4ee40039c0 2024-03-23T15:47:38.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee400c1f0 con 0x7f4ee40047c0 2024-03-23T15:47:38.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.870+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee400c3b0 con 0x7f4ee40052a0 2024-03-23T15:47:38.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.873+0000 7f4efbfff640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee40039c0 0x7f4ee4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.875+0000 7f4efb7fe640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee40047c0 0x7f4ee4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.876+0000 7f4efbfff640 1 -- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40052a0 msgr2=0x7f4ee400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.876+0000 7f4efbfff640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40052a0 0x7f4ee400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.876+0000 7f4efbfff640 1 -- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee40047c0 msgr2=0x7f4ee4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.876+0000 7f4efbfff640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee40047c0 0x7f4ee4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.876+0000 7f4efbfff640 1 -- 172.21.15.129:0/1769066086 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee4003520 con 0x7f4ee40039c0 2024-03-23T15:47:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.876+0000 7f4f00852640 1 -- 172.21.15.129:0/1769066086 reap_dead start 2024-03-23T15:47:38.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.878+0000 7f4efbfff640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee40039c0 0x7f4ee4003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4eec006fd0 tx=0x7f4ee40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b16ecd32fe49ad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:38.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.884+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec032600 con 0x7f4ee40039c0 2024-03-23T15:47:38.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.884+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eec02e050 con 0x7f4ee40039c0 2024-03-23T15:47:38.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.884+0000 7f4efaffd640 1 -- 172.21.15.129:0/1769066086 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0329c0 con 0x7f4ee40039c0 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 -- 172.21.15.129:0/1769066086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee40039c0 msgr2=0x7f4ee4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee40039c0 0x7f4ee4003dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4eec006fd0 tx=0x7f4ee40057f0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 -- 172.21.15.129:0/1769066086 shutdown_connections 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 --2- 172.21.15.129:0/1769066086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee40039c0 0x7f4ee4003dc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 -- 172.21.15.129:0/1769066086 >> 172.21.15.129:0/1769066086 conn(0x7f4efc0c8370 msgr2=0x7f4efc0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 -- 172.21.15.129:0/1769066086 shutdown_connections 2024-03-23T15:47:38.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.885+0000 7f4f022b5640 1 -- 172.21.15.129:0/1769066086 wait complete. 2024-03-23T15:47:38.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.887+0000 7f4f022b5640 1 Processor -- start 2024-03-23T15:47:38.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.887+0000 7f4f022b5640 1 -- start start 2024-03-23T15:47:38.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f022b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40039c0 0x7f4efc1456e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f022b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 0x7f4efc158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f022b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4efc145f30 0x7f4efc1f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f022b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4efc159550 con 0x7f4efc0c6c80 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f022b5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4efc1554c0 con 0x7f4efc145f30 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f022b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4efc156d90 con 0x7f4ee40039c0 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4efb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 0x7f4efc158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.889+0000 7f4f00852640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4efc145f30 0x7f4efc1f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.890+0000 7f4efb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 0x7f4efc158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58864/0 (socket says 172.21.15.129:58864) 2024-03-23T15:47:38.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.890+0000 7f4efb7fe640 1 -- 172.21.15.129:0/730752489 learned_addr learned my addr 172.21.15.129:0/730752489 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:38.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.894+0000 7f4efb7fe640 1 -- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40039c0 msgr2=0x7f4efc1456e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.894+0000 7f4efb7fe640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40039c0 0x7f4efc1456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.894+0000 7f4efb7fe640 1 -- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4efc145f30 msgr2=0x7f4efc1f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:38.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.894+0000 7f4efb7fe640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4efc145f30 0x7f4efc1f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:38.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.894+0000 7f4efb7fe640 1 -- 172.21.15.129:0/730752489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4efc145c20 con 0x7f4efc0c6c80 2024-03-23T15:47:38.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.897+0000 7f4efb7fe640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 0x7f4efc158390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000e540 tx=0x7f4ef0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:38.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.902+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00052e0 con 0x7f4efc0c6c80 2024-03-23T15:47:38.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.902+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef0005460 con 0x7f4efc0c6c80 2024-03-23T15:47:38.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.902+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0003830 con 0x7f4efc0c6c80 2024-03-23T15:47:38.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.902+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4efc1567d0 con 0x7f4efc0c6c80 2024-03-23T15:47:38.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.902+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4efc154e70 con 0x7f4efc0c6c80 2024-03-23T15:47:38.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:38.903+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4efc0c3390 con 0x7f4efc0c6c80 2024-03-23T15:47:39.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.167+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ef0003990 con 0x7f4efc0c6c80 2024-03-23T15:47:39.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.167+0000 7f4ef8ff9640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4ee806b810 0x7f4ee806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.168+0000 7f4efbfff640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4ee806b810 0x7f4ee806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.168+0000 7f4efbfff640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4ee806b810 0x7f4ee806dcd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4eec034000 tx=0x7f4eec02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.189+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4ef00598c0 con 0x7f4efc0c6c80 2024-03-23T15:47:39.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.190+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ef0055460 con 0x7f4efc0c6c80 2024-03-23T15:47:39.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.413+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4efc0c4fe0 con 0x7f4efc0c6c80 2024-03-23T15:47:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.469+0000 7f4ef8ff9640 1 -- 172.21.15.129:0/730752489 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f4ef0058b20 con 0x7f4efc0c6c80 2024-03-23T15:47:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:39.472 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.471+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4ee806b810 msgr2=0x7f4ee806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.471+0000 7f4f022b5640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4ee806b810 0x7f4ee806dcd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4eec034000 tx=0x7f4eec02f000 comp rx=0 tx=0).stop 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.471+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 msgr2=0x7f4efc158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.471+0000 7f4f022b5640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 0x7f4efc158390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000e540 tx=0x7f4ef0001e70 comp rx=0 tx=0).stop 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 shutdown_connections 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4ee806b810 0x7f4ee806dcd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4efc145f30 0x7f4efc1f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4efc0c6c80 0x7f4efc158390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 --2- 172.21.15.129:0/730752489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ee40039c0 0x7f4efc1456e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 >> 172.21.15.129:0/730752489 conn(0x7f4efc0c8370 msgr2=0x7f4efc0c8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 shutdown_connections 2024-03-23T15:47:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.472+0000 7f4f022b5640 1 -- 172.21.15.129:0/730752489 wait complete. 2024-03-23T15:47:39.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-23T15:47:39.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-23T15:47:39.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-23T15:47:39.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-23T15:47:39.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-23T15:47:39.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-23T15:47:39.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-23T15:47:39.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-23T15:47:39.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.675+0000 7f5d11e42640 1 Processor -- start 2024-03-23T15:47:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.675+0000 7f5d11e42640 1 -- start start 2024-03-23T15:47:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f5d11e42640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5d040075f0 0x7f5d040452f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f5d11e42640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d040fff20 0x7f5d041003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f5d11e42640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d04007160 con 0x7f5d0404b0d0 2024-03-23T15:47:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f5d11e42640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5d0404c790 con 0x7f5d040075f0 2024-03-23T15:47:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f5d11e42640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0404ca70 con 0x7f5d040fff20 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f0513fff640 1 Processor -- start 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.676+0000 7f0513fff640 1 -- start start 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.677+0000 7f0513fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f050c04bb80 0x7f050c0efb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.677+0000 7f0513fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 0x7f050c04bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.677+0000 7f0513fff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f050c011100 con 0x7f050c039f90 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.677+0000 7f0513fff640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f050c0076d0 con 0x7f050c04bb80 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.677+0000 7f0513fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f050c04c9e0 con 0x7f050c0ebd00 2024-03-23T15:47:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f5d0b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5d040075f0 0x7f5d040452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f5d0affd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f5d0404b0d0 0x7f5d040f6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36534/0 (socket says 172.21.15.129:36534) 2024-03-23T15:47:39.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f5d0affd640 1 -- 172.21.15.129:0/2265288159 learned_addr learned my addr 172.21.15.129:0/2265288159 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:39.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f05127fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 0x7f050c04bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f05127fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 0x7f050c04bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35688/0 (socket says 172.21.15.129:35688) 2024-03-23T15:47:39.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f05127fc640 1 -- 172.21.15.129:0/3564086521 learned_addr learned my addr 172.21.15.129:0/3564086521 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.680+0000 7f0512ffd640 1 --2- 172.21.15.129:0/3564086521 >> v2:172.21.15.129:3300/0 conn(0x7f050c04bb80 0x7f050c0efb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.678+0000 7f5d10be0640 1 --2- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d040fff20 0x7f5d041003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.683+0000 7f5d0b7fe640 1 -- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d040fff20 msgr2=0x7f5d041003b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.683+0000 7f5d0b7fe640 1 --2- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d040fff20 0x7f5d041003b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.683+0000 7f5d0b7fe640 1 -- 172.21.15.129:0/2265288159 >> v1:172.21.15.129:6789/0 conn(0x7f5d0404b0d0 legacy=0x7f5d040f6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.683+0000 7f5d0b7fe640 1 -- 172.21.15.129:0/2265288159 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d04006b10 con 0x7f5d040075f0 2024-03-23T15:47:39.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.687+0000 7f8713b25640 1 Processor -- start 2024-03-23T15:47:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.687+0000 7f8713b25640 1 -- start start 2024-03-23T15:47:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f8713b25640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c1589f0 0x7f870c06fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f8713b25640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c071ca0 con 0x7f870c0724a0 2024-03-23T15:47:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f8713b25640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c071f80 con 0x7f870c0c7f40 2024-03-23T15:47:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f8713b25640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f870c06f110 con 0x7f870c1589f0 2024-03-23T15:47:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.690+0000 7f87120c2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f870c0c7f40 0x7f870c168fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33290/0 (socket says 172.21.15.129:33290) 2024-03-23T15:47:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.690+0000 7f87120c2640 1 -- 172.21.15.129:0/478027275 learned_addr learned my addr 172.21.15.129:0/478027275 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.692+0000 7f5d0b7fe640 1 --2- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 0x7f5d040452f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c062cd0 tx=0x7f5d0c062ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189b744e29742516 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.694+0000 7f87118c1640 1 --2- 172.21.15.129:0/478027275 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c1589f0 0x7f870c06fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.694+0000 7fc765699640 1 Processor -- start 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.695+0000 7fc765699640 1 -- start start 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7fc765699640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 0x7fc760156820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7fc765699640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc76006db80 0x7fc760156d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7fc765699640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc7600c68d0 0x7fc76006cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7fc765699640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc760071b80 con 0x7fc7600c68d0 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7fc765699640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc76015a2e0 con 0x7fc76006db80 2024-03-23T15:47:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7fc765699640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc7600715e0 con 0x7fc760150540 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.695+0000 7f9d0e6b1640 1 Processor -- start 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.695+0000 7f9d0e6b1640 1 -- start start 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7f9d0e6b1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9d080d0000 0x7f9d080d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7f9d0e6b1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080ca4a0 0x7f9d080ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7f9d0e6b1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9d080cc990 0x7f9d080cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7f9d0e6b1640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9d080d0920 con 0x7f9d080cc990 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7f9d0e6b1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9d08154ac0 con 0x7f9d080d0000 2024-03-23T15:47:39.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.696+0000 7f9d0e6b1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9d080cd360 con 0x7f9d080ca4a0 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.697+0000 7f87108bf640 1 -- 172.21.15.129:0/478027275 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372471746 0 0) 0x7f870c071f80 con 0x7f870c0c7f40 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.697+0000 7f87108bf640 1 -- 172.21.15.129:0/478027275 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f8003520 con 0x7f870c0c7f40 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f05127fc640 1 -- 172.21.15.129:0/3564086521 >> v2:172.21.15.129:3300/0 conn(0x7f050c04bb80 msgr2=0x7f050c0efb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f05127fc640 1 --2- 172.21.15.129:0/3564086521 >> v2:172.21.15.129:3300/0 conn(0x7f050c04bb80 0x7f050c0efb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f05127fc640 1 -- 172.21.15.129:0/3564086521 >> v1:172.21.15.157:6789/0 conn(0x7f050c039f90 legacy=0x7f050c0e8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.689+0000 7f05127fc640 1 -- 172.21.15.129:0/3564086521 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c04ccc0 con 0x7f050c0ebd00 2024-03-23T15:47:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.694+0000 7f05127fc640 1 --2- 172.21.15.129:0/3564086521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 0x7f050c04bf60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0514064730 tx=0x7f0514064ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f119d6c3ea6cadb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.697+0000 7f9d0cc4e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9d080cc990 0x7f9d080cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.698+0000 7f9d0cc4e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9d080cc990 0x7f9d080cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58906/0 (socket says 172.21.15.129:58906) 2024-03-23T15:47:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.698+0000 7f9d0cc4e640 1 -- 172.21.15.129:0/2451038906 learned_addr learned my addr 172.21.15.129:0/2451038906 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.698+0000 7f9d07fff640 1 --2- 172.21.15.129:0/2451038906 >> v2:172.21.15.157:3300/0 conn(0x7f9d080d0000 0x7f9d080d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.698+0000 7fc764e98640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc7600c68d0 0x7fc76006cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.699+0000 7f5d0a7fc640 1 -- 172.21.15.129:0/2265288159 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c09c5d0 con 0x7f5d040075f0 2024-03-23T15:47:39.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.699+0000 7f5d0a7fc640 1 -- 172.21.15.129:0/2265288159 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0c09a050 con 0x7f5d040075f0 2024-03-23T15:47:39.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.699+0000 7f5d0a7fc640 1 -- 172.21.15.129:0/2265288159 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c09ca30 con 0x7f5d040075f0 2024-03-23T15:47:39.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 -- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 msgr2=0x7f5d040452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 --2- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 0x7f5d040452f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c062cd0 tx=0x7f5d0c062ee0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 -- 172.21.15.129:0/2265288159 shutdown_connections 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 --2- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d040fff20 0x7f5d041003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 --2- 172.21.15.129:0/2265288159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 0x7f5d040452f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 -- 172.21.15.129:0/2265288159 >> 172.21.15.129:0/2265288159 conn(0x7f5d04050350 msgr2=0x7f5d04050760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 -- 172.21.15.129:0/2265288159 shutdown_connections 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7f5d11e42640 1 -- 172.21.15.129:0/2265288159 wait complete. 2024-03-23T15:47:39.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.700+0000 7fc757fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc76006db80 0x7fc760156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.701+0000 7f8f3d489640 1 Processor -- start 2024-03-23T15:47:39.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.701+0000 7f8f3d489640 1 -- start start 2024-03-23T15:47:39.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.701+0000 7f87118c1640 1 -- 172.21.15.129:0/478027275 >> v1:172.21.15.157:6789/0 conn(0x7f870c0c7f40 legacy=0x7f870c168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.701+0000 7f87118c1640 1 -- 172.21.15.129:0/478027275 >> v1:172.21.15.129:6789/0 conn(0x7f870c0724a0 legacy=0x7f870c0c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.701+0000 7f87118c1640 1 -- 172.21.15.129:0/478027275 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c06f3f0 con 0x7f870c1589f0 2024-03-23T15:47:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.702+0000 7f8f3d489640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 0x7f8f30007bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.702+0000 7f8f3d489640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f300e2f50 con 0x7f8f300d90a0 2024-03-23T15:47:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.702+0000 7f8f3d489640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f30009280 con 0x7f8f3004c330 2024-03-23T15:47:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.702+0000 7f8f3d489640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f300e2360 con 0x7f8f30046510 2024-03-23T15:47:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.702+0000 7f5d11e42640 1 Processor -- start 2024-03-23T15:47:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.702+0000 7f5d11e42640 1 -- start start 2024-03-23T15:47:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f0511ffb640 1 -- 172.21.15.129:0/3564086521 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05140ac5d0 con 0x7f050c0ebd00 2024-03-23T15:47:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f0511ffb640 1 -- 172.21.15.129:0/3564086521 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05140aa050 con 0x7f050c0ebd00 2024-03-23T15:47:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f0511ffb640 1 -- 172.21.15.129:0/3564086521 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05140aca30 con 0x7f050c0ebd00 2024-03-23T15:47:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7fc757fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc76006db80 0x7fc760156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41118/0 (socket says 172.21.15.129:41118) 2024-03-23T15:47:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7fc757fff640 1 -- 172.21.15.129:0/3443303716 learned_addr learned my addr 172.21.15.129:0/3443303716 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f5d11e42640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 0x7f5d040f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f5d11e42640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d0404b0d0 0x7f5d040f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f0513fff640 1 -- 172.21.15.129:0/3564086521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 msgr2=0x7f050c04bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.703+0000 7f0513fff640 1 --2- 172.21.15.129:0/3564086521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 0x7f050c04bf60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0514064730 tx=0x7f0514064ea0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f0513fff640 1 -- 172.21.15.129:0/3564086521 shutdown_connections 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f0513fff640 1 --2- 172.21.15.129:0/3564086521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c0ebd00 0x7f050c04bf60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f0513fff640 1 --2- 172.21.15.129:0/3564086521 >> v2:172.21.15.129:3300/0 conn(0x7f050c04bb80 0x7f050c0efb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f0513fff640 1 -- 172.21.15.129:0/3564086521 >> 172.21.15.129:0/3564086521 conn(0x7f050c0061a0 msgr2=0x7f050c0065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f0513fff640 1 -- 172.21.15.129:0/3564086521 shutdown_connections 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f5d11e42640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040fff20 0x7f5d040f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f0513fff640 1 -- 172.21.15.129:0/3564086521 wait complete. 2024-03-23T15:47:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f5d11e42640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d040fcc00 con 0x7f5d040fff20 2024-03-23T15:47:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f5d11e42640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5d040fd130 con 0x7f5d040075f0 2024-03-23T15:47:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.704+0000 7f5d11e42640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5d040fce50 con 0x7f5d0404b0d0 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f9d0cc4e640 1 -- 172.21.15.129:0/2451038906 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080ca4a0 msgr2=0x7f9d080ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f9d0cc4e640 1 --2- 172.21.15.129:0/2451038906 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080ca4a0 0x7f9d080ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f9d0cc4e640 1 -- 172.21.15.129:0/2451038906 >> v2:172.21.15.157:3300/0 conn(0x7f9d080d0000 msgr2=0x7f9d080d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f9d0cc4e640 1 --2- 172.21.15.129:0/2451038906 >> v2:172.21.15.157:3300/0 conn(0x7f9d080d0000 0x7f9d080d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f9d0cc4e640 1 -- 172.21.15.129:0/2451038906 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0815b270 con 0x7f9d080cc990 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f0513fff640 1 Processor -- start 2024-03-23T15:47:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.705+0000 7f0513fff640 1 -- start start 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f87118c1640 1 --2- 172.21.15.129:0/478027275 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c1589f0 0x7f870c06fbf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0097d0 tx=0x7f86fc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bd0d374e6946a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f0513fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c039f90 0x7f050c04f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f0513fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c04bb80 0x7f050c04fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f0513fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c00aff0 0x7f050c177b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f0513fff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f050c00ad50 con 0x7f050c04bb80 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f0513fff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f050c047930 con 0x7f050c00aff0 2024-03-23T15:47:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.706+0000 7f0513fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f050c1773a0 con 0x7f050c039f90 2024-03-23T15:47:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.707+0000 7fc75ffff640 1 --2- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 0x7fc760156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.708+0000 7f5d10be0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040fff20 0x7f5d040f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.708+0000 7f5d10be0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040fff20 0x7f5d040f8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58910/0 (socket says 172.21.15.129:58910) 2024-03-23T15:47:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.708+0000 7f5d10be0640 1 -- 172.21.15.129:0/3580529869 learned_addr learned my addr 172.21.15.129:0/3580529869 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.709+0000 7f50eceed640 1 Processor -- start 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.709+0000 7f50eceed640 1 -- start start 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7fc757fff640 1 -- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 msgr2=0x7fc760156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7fc757fff640 1 --2- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 0x7fc760156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7fc757fff640 1 -- 172.21.15.129:0/3443303716 >> v2:172.21.15.129:3300/0 conn(0x7fc7600c68d0 msgr2=0x7fc76006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7fc757fff640 1 --2- 172.21.15.129:0/3443303716 >> v2:172.21.15.129:3300/0 conn(0x7fc7600c68d0 0x7fc76006cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7fc757fff640 1 -- 172.21.15.129:0/3443303716 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76015a5a0 con 0x7fc76006db80 2024-03-23T15:47:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7f8f3cc88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 0x7f8f30007bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7f50eceed640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f50e814d890 0x7f50e8072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7f50eceed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e80c7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7f50eceed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50e80c4b20 con 0x7f50e80c7860 2024-03-23T15:47:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7f50eceed640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f50e80732b0 con 0x7f50e814d890 2024-03-23T15:47:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.710+0000 7f50eceed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f50e80734b0 con 0x7f50e80c6b50 2024-03-23T15:47:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.712+0000 7f8f3cc88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 0x7f8f30007bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35730/0 (socket says 172.21.15.129:35730) 2024-03-23T15:47:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.712+0000 7f8f3cc88640 1 -- 172.21.15.129:0/604166158 learned_addr learned my addr 172.21.15.129:0/604166158 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.712+0000 7f0512ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c039f90 0x7f050c04f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.713+0000 7f5d0affd640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d0404b0d0 0x7f5d040f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.715+0000 7f0512ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c039f90 0x7f050c04f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35750/0 (socket says 172.21.15.129:35750) 2024-03-23T15:47:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.715+0000 7f0512ffd640 1 -- 172.21.15.129:0/1202698842 learned_addr learned my addr 172.21.15.129:0/1202698842 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.715+0000 7f5d0b7fe640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 0x7f5d040f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.716+0000 7fc757fff640 1 --2- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc76006db80 0x7fc760156d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7600cc680 tx=0x7fc74802dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dc3a5c138604e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.716+0000 7f5d10be0640 1 -- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d0404b0d0 msgr2=0x7f5d040f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.716+0000 7f5d10be0640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d0404b0d0 0x7f5d040f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.716+0000 7f5d10be0640 1 -- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 msgr2=0x7f5d040f9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.716+0000 7f5d10be0640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d040075f0 0x7f5d040f9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.716+0000 7f5d10be0640 1 -- 172.21.15.129:0/3580529869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c06a250 con 0x7f5d040fff20 2024-03-23T15:47:39.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.719+0000 7f50e659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f50e814d890 0x7f50e8072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.720+0000 7f05127fc640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c04bb80 0x7f050c04fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.720+0000 7f05137fe640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c00aff0 0x7f050c177b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.722+0000 7f0512ffd640 1 -- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c00aff0 msgr2=0x7f050c177b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.722+0000 7f0512ffd640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c00aff0 0x7f050c177b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.722+0000 7f0512ffd640 1 -- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c04bb80 msgr2=0x7f050c04fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.722+0000 7f0512ffd640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c04bb80 0x7f050c04fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.722+0000 7f0512ffd640 1 -- 172.21.15.129:0/1202698842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c047b80 con 0x7f050c039f90 2024-03-23T15:47:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.723+0000 7f5d10be0640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040fff20 0x7f5d040f8640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800d830 tx=0x7f5cf800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.723+0000 7f50e659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f50e814d890 0x7f50e8072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41152/0 (socket says 172.21.15.129:41152) 2024-03-23T15:47:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.723+0000 7f50e659c640 1 -- 172.21.15.129:0/1080980357 learned_addr learned my addr 172.21.15.129:0/1080980357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.724+0000 7f9d0cc4e640 1 --2- 172.21.15.129:0/2451038906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 0x7f9d080cce20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d080d0bd0 tx=0x7f9cfc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a890549a95063e41 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.725+0000 7f50dffff640 1 --2- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e80c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.726+0000 7fe72b39a640 1 Processor -- start 2024-03-23T15:47:39.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.727+0000 7fe72b39a640 1 -- start start 2024-03-23T15:47:39.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.728+0000 7fe72b39a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe72406b780 0x7fe724073240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.728+0000 7fe72b39a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7240c9dd0 con 0x7fe72415b0b0 2024-03-23T15:47:39.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.728+0000 7fe72b39a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72415a510 con 0x7fe72406b3a0 2024-03-23T15:47:39.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.728+0000 7fe72b39a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe724073780 con 0x7fe72406b780 2024-03-23T15:47:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.730+0000 7fc75f7fe640 1 -- 172.21.15.129:0/3443303716 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74802e960 con 0x7fc76006db80 2024-03-23T15:47:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.730+0000 7fc75f7fe640 1 -- 172.21.15.129:0/3443303716 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74802eac0 con 0x7fc76006db80 2024-03-23T15:47:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.730+0000 7fc75f7fe640 1 -- 172.21.15.129:0/3443303716 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74802ee50 con 0x7fc76006db80 2024-03-23T15:47:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.731+0000 7fc765699640 1 -- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc76006db80 msgr2=0x7fc760156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.731+0000 7fc765699640 1 --2- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc76006db80 0x7fc760156d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7600cc680 tx=0x7fc74802dbb0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 -- 172.21.15.129:0/3443303716 shutdown_connections 2024-03-23T15:47:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 --2- 172.21.15.129:0/3443303716 >> v2:172.21.15.129:3300/0 conn(0x7fc7600c68d0 0x7fc76006cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 --2- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc76006db80 0x7fc760156d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 --2- 172.21.15.129:0/3443303716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 0x7fc760156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 -- 172.21.15.129:0/3443303716 >> 172.21.15.129:0/3443303716 conn(0x7fc76006b8f0 msgr2=0x7fc7600cb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 -- 172.21.15.129:0/3443303716 shutdown_connections 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7fc765699640 1 -- 172.21.15.129:0/3443303716 wait complete. 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f8f3cc88640 1 -- 172.21.15.129:0/604166158 >> v1:172.21.15.157:6789/0 conn(0x7f8f3004c330 legacy=0x7f8f300074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f8f3cc88640 1 -- 172.21.15.129:0/604166158 >> v1:172.21.15.129:6789/0 conn(0x7f8f300d90a0 legacy=0x7f8f3004bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f8f3cc88640 1 -- 172.21.15.129:0/604166158 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f300d9480 con 0x7f8f30046510 2024-03-23T15:47:39.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f0512ffd640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c039f90 0x7f050c04f9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f050800e9e0 tx=0x7f050800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f50e659c640 1 -- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 msgr2=0x7f50e80c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f50e659c640 1 --2- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e80c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.732+0000 7f50e659c640 1 -- 172.21.15.129:0/1080980357 >> v1:172.21.15.129:6789/0 conn(0x7f50e80c7860 legacy=0x7f50e8157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:39.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.733+0000 7f50e659c640 1 -- 172.21.15.129:0/1080980357 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e80c3f50 con 0x7f50e814d890 2024-03-23T15:47:39.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.734+0000 7fc765699640 1 Processor -- start 2024-03-23T15:47:39.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.734+0000 7fc765699640 1 -- start start 2024-03-23T15:47:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.735+0000 7fc765699640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7600c68d0 0x7fc7600c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.735+0000 7fc765699640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 0x7fc76006cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.735+0000 7fc765699640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc760148e10 0x7fc7601492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.735+0000 7fc765699640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc760159180 con 0x7fc7600c68d0 2024-03-23T15:47:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.735+0000 7fc765699640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc760149800 con 0x7fc760148e10 2024-03-23T15:47:39.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.735+0000 7fc765699640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc7601efc60 con 0x7fc760150540 2024-03-23T15:47:39.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.737+0000 7f9d06ffd640 1 -- 172.21.15.129:0/2451038906 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc030850 con 0x7f9d080cc990 2024-03-23T15:47:39.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.737+0000 7f9d06ffd640 1 -- 172.21.15.129:0/2451038906 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cfc0309b0 con 0x7f9d080cc990 2024-03-23T15:47:39.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.737+0000 7f9d06ffd640 1 -- 172.21.15.129:0/2451038906 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc030d40 con 0x7f9d080cc990 2024-03-23T15:47:39.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/2451038906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 msgr2=0x7f9d080cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/2451038906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 0x7f9d080cce20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d080d0bd0 tx=0x7f9cfc02b990 comp rx=0 tx=0).stop 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/2451038906 shutdown_connections 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/2451038906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 0x7f9d080cce20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/2451038906 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080ca4a0 0x7f9d080ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/2451038906 >> v2:172.21.15.157:3300/0 conn(0x7f9d080d0000 0x7f9d080d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/2451038906 >> 172.21.15.129:0/2451038906 conn(0x7f9d08061f40 msgr2=0x7f9d08062350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/2451038906 shutdown_connections 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/2451038906 wait complete. 2024-03-23T15:47:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800b6c0 con 0x7f5d040fff20 2024-03-23T15:47:39.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7fc75ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7600c68d0 0x7fc7600c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d040fc4e0 con 0x7f5d040fff20 2024-03-23T15:47:39.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf800b840 con 0x7f5d040fff20 2024-03-23T15:47:39.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7fc75ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7600c68d0 0x7fc7600c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58928/0 (socket says 172.21.15.129:58928) 2024-03-23T15:47:39.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7fc75ffff640 1 -- 172.21.15.129:0/1452281615 learned_addr learned my addr 172.21.15.129:0/1452281615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.739+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d040f8120 con 0x7f5d040fff20 2024-03-23T15:47:39.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.740+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800bbf0 con 0x7f5d040fff20 2024-03-23T15:47:39.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.740+0000 7f9d0e6b1640 1 Processor -- start 2024-03-23T15:47:39.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.740+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d04051000 con 0x7f5d040fff20 2024-03-23T15:47:39.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 -- start start 2024-03-23T15:47:39.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f50e659c640 1 --2- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e814d890 0x7f50e8072d70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009870 tx=0x7f50d002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bdacec1a50d78b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d080ca4a0 0x7f9d080b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 0x7f9d081599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080d0000 0x7f9d081eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d081f3e40 con 0x7f9d080cc990 2024-03-23T15:47:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9d080b3390 con 0x7f9d080ca4a0 2024-03-23T15:47:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.741+0000 7f9d0e6b1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9d081f3550 con 0x7f9d080d0000 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.742+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0508009670 con 0x7f050c039f90 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.742+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f050800c9d0 con 0x7f050c039f90 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.742+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050c181e40 con 0x7f050c039f90 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.742+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050800cd80 con 0x7f050c039f90 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.742+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f050c182120 con 0x7f050c039f90 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.743+0000 7f8f3cc88640 1 --2- 172.21.15.129:0/604166158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 0x7f8f30007bc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f38050d70 tx=0x7f8f38062ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4c433e0e7468cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.743+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f050c0f00c0 con 0x7f050c039f90 2024-03-23T15:47:39.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.749+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d080ca4a0 0x7f9d080b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.749+0000 7fc764e98640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc760148e10 0x7fc7601492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.749+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d080ca4a0 0x7f9d080b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41194/0 (socket says 172.21.15.129:41194) 2024-03-23T15:47:39.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.749+0000 7f9d07fff640 1 -- 172.21.15.129:0/3655483713 learned_addr learned my addr 172.21.15.129:0/3655483713 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fc75ffff640 1 -- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 msgr2=0x7fc76006cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fc75ffff640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc760150540 0x7fc76006cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fc75ffff640 1 -- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc760148e10 msgr2=0x7fc7601492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fc75ffff640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc760148e10 0x7fc7601492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fc75ffff640 1 -- 172.21.15.129:0/1452281615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7480094d0 con 0x7fc7600c68d0 2024-03-23T15:47:39.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fe728935640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fe72415b0b0 0x7fe724154040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36574/0 (socket says 172.21.15.129:36574) 2024-03-23T15:47:39.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.755+0000 7fe728935640 1 -- 172.21.15.129:0/3517955476 learned_addr learned my addr 172.21.15.129:0/3517955476 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:39.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.756+0000 7fe729937640 1 --2- 172.21.15.129:0/3517955476 >> v2:172.21.15.157:3300/0 conn(0x7fe72406b780 0x7fe724073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.757+0000 7f9d077fe640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 0x7f9d081599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.759+0000 7fc75ffff640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7600c68d0 0x7fc7600c9670 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc75000a7a0 tx=0x7fc75000ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.760+0000 7f9d07fff640 1 -- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080d0000 msgr2=0x7f9d081eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.760+0000 7f9d07fff640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d080d0000 0x7f9d081eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.760+0000 7f9d07fff640 1 -- 172.21.15.129:0/3655483713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 msgr2=0x7f9d081599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.760+0000 7f9d07fff640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d080cc990 0x7f9d081599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.760+0000 7f9d07fff640 1 -- 172.21.15.129:0/3655483713 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cfc0094d0 con 0x7f9d080ca4a0 2024-03-23T15:47:39.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.766+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0508013470 con 0x7f050c039f90 2024-03-23T15:47:39.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.766+0000 7f04fb7fe640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f050006b860 0x7f050006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.767+0000 7f05127fc640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f050006b860 0x7f050006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.767+0000 7f05127fc640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f050006b860 0x7f050006dd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f050c04cb20 tx=0x7f05140ac000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.768+0000 7f9d07fff640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d080ca4a0 0x7f9d080b2d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800d800 tx=0x7f9cf800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7fe729937640 1 -- 172.21.15.129:0/3517955476 >> v1:172.21.15.157:6789/0 conn(0x7fe72406b3a0 legacy=0x7fe72415a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7fe729937640 1 -- 172.21.15.129:0/3517955476 >> v1:172.21.15.129:6789/0 conn(0x7fe72415b0b0 legacy=0x7fe724154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7fe729937640 1 -- 172.21.15.129:0/3517955476 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7240c73f0 con 0x7fe72406b780 2024-03-23T15:47:39.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0508059660 con 0x7f050c039f90 2024-03-23T15:47:39.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7f04fb7fe640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f0500073a60 0x7f0500075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 --> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0500076530 con 0x7f0500073a60 2024-03-23T15:47:39.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7f05137fe640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f0500073a60 0x7f0500075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.772+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f050800cee0 con 0x7f050c039f90 2024-03-23T15:47:39.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.773+0000 7f05137fe640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f0500073a60 0x7f0500075e80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.773+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== osd.1 v2:172.21.15.129:6816/168045261 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0500076530 con 0x7f0500073a60 2024-03-23T15:47:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.775+0000 7f50e5d9b640 1 -- 172.21.15.129:0/1080980357 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0031640 con 0x7f50e814d890 2024-03-23T15:47:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.775+0000 7f50e5d9b640 1 -- 172.21.15.129:0/1080980357 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d00317a0 con 0x7f50e814d890 2024-03-23T15:47:39.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50e5d9b640 1 -- 172.21.15.129:0/1080980357 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0031640 con 0x7f50e814d890 2024-03-23T15:47:39.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 -- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e814d890 msgr2=0x7f50e8072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 --2- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e814d890 0x7f50e8072d70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009870 tx=0x7f50d002b8f0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 -- 172.21.15.129:0/1080980357 shutdown_connections 2024-03-23T15:47:39.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 --2- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e80c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 --2- 172.21.15.129:0/1080980357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e814d890 0x7f50e8072d70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 -- 172.21.15.129:0/1080980357 >> 172.21.15.129:0/1080980357 conn(0x7f50e806c240 msgr2=0x7f50e806c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f87108bf640 1 -- 172.21.15.129:0/478027275 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc030610 con 0x7f870c1589f0 2024-03-23T15:47:39.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f87108bf640 1 -- 172.21.15.129:0/478027275 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86fc030770 con 0x7f870c1589f0 2024-03-23T15:47:39.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.776+0000 7f50eceed640 1 -- 172.21.15.129:0/1080980357 shutdown_connections 2024-03-23T15:47:39.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f87108bf640 1 -- 172.21.15.129:0/478027275 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc030b00 con 0x7f870c1589f0 2024-03-23T15:47:39.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f50eceed640 1 -- 172.21.15.129:0/1080980357 wait complete. 2024-03-23T15:47:39.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 -- 172.21.15.129:0/478027275 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c1589f0 msgr2=0x7f870c06fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 --2- 172.21.15.129:0/478027275 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c1589f0 0x7f870c06fbf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0097d0 tx=0x7f86fc02b960 comp rx=0 tx=0).stop 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 -- 172.21.15.129:0/478027275 shutdown_connections 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 --2- 172.21.15.129:0/478027275 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c1589f0 0x7f870c06fbf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 -- 172.21.15.129:0/478027275 >> 172.21.15.129:0/478027275 conn(0x7f870c0cb910 msgr2=0x7f870c0cbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 -- 172.21.15.129:0/478027275 shutdown_connections 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.777+0000 7f8713b25640 1 -- 172.21.15.129:0/478027275 wait complete. 2024-03-23T15:47:39.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.779+0000 7f50eceed640 1 Processor -- start 2024-03-23T15:47:39.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.779+0000 7f50eceed640 1 -- start start 2024-03-23T15:47:39.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.779+0000 7f8713b25640 1 Processor -- start 2024-03-23T15:47:39.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.779+0000 7fe729937640 1 --2- 172.21.15.129:0/3517955476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72406b780 0x7fe724073240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009dc0 tx=0x7fe71c02bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8a64df219f8bc42 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.780+0000 7f8713b25640 1 -- start start 2024-03-23T15:47:39.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.780+0000 7f50eceed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e8146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.780+0000 7f50eceed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e80c7860 0x7f50e81ef610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f50eceed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e814d890 0x7f50e81560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f50eceed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f50e81ef1f0 con 0x7f50e814d890 2024-03-23T15:47:39.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f50eceed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f50e81ec930 con 0x7f50e80c7860 2024-03-23T15:47:39.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f50eceed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f50e8147460 con 0x7f50e80c6b50 2024-03-23T15:47:39.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f8713b25640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870c0724a0 0x7f870c136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f8713b25640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c0c7f40 0x7f870c1366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f8713b25640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f870c1589f0 0x7f870c15a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f8713b25640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f870c158440 con 0x7f870c0724a0 2024-03-23T15:47:39.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f8713b25640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f870c153560 con 0x7f870c1589f0 2024-03-23T15:47:39.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.781+0000 7f8713b25640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f870c153820 con 0x7f870c0c7f40 2024-03-23T15:47:39.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.784+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800b6c0 con 0x7f9d080ca4a0 2024-03-23T15:47:39.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.784+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf800b820 con 0x7f9d080ca4a0 2024-03-23T15:47:39.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.784+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800bbb0 con 0x7f9d080ca4a0 2024-03-23T15:47:39.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.784+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d081f37f0 con 0x7f9d080ca4a0 2024-03-23T15:47:39.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.784+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d081f42d0 con 0x7f9d080ca4a0 2024-03-23T15:47:39.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.784+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5cf801e460 con 0x7f5d040fff20 2024-03-23T15:47:39.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.785+0000 7f5cf7fff640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5cf006b810 0x7f5cf006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.785+0000 7f87110c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c0c7f40 0x7f870c1366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.785+0000 7f5d0b7fe640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5cf006b810 0x7f5cf006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.785+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d0815acb0 con 0x7f9d080ca4a0 2024-03-23T15:47:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.786+0000 7f5d0b7fe640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5cf006b810 0x7f5cf006dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400dd30 tx=0x7f5d0c09b000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.788+0000 7f8f36ffd640 1 -- 172.21.15.129:0/604166158 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f380af420 con 0x7f8f30046510 2024-03-23T15:47:39.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.788+0000 7f50e659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e8146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.788+0000 7f8f36ffd640 1 -- 172.21.15.129:0/604166158 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f3806d040 con 0x7f8f30046510 2024-03-23T15:47:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.788+0000 7f8f36ffd640 1 -- 172.21.15.129:0/604166158 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f380af8c0 con 0x7f8f30046510 2024-03-23T15:47:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.788+0000 7f50e659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e8146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35780/0 (socket says 172.21.15.129:35780) 2024-03-23T15:47:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.788+0000 7f50e659c640 1 -- 172.21.15.129:0/867272251 learned_addr learned my addr 172.21.15.129:0/867272251 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 -- 172.21.15.129:0/604166158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 msgr2=0x7f8f30007bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 --2- 172.21.15.129:0/604166158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 0x7f8f30007bc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f38050d70 tx=0x7f8f38062ee0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 -- 172.21.15.129:0/604166158 shutdown_connections 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 --2- 172.21.15.129:0/604166158 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f30046510 0x7f8f30007bc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 -- 172.21.15.129:0/604166158 >> 172.21.15.129:0/604166158 conn(0x7f8f3004dd20 msgr2=0x7f8f3004e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 -- 172.21.15.129:0/604166158 shutdown_connections 2024-03-23T15:47:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f8f3d489640 1 -- 172.21.15.129:0/604166158 wait complete. 2024-03-23T15:47:39.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.789+0000 7f87110c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c0c7f40 0x7f870c1366c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35782/0 (socket says 172.21.15.129:35782) 2024-03-23T15:47:39.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.790+0000 7f87110c0640 1 -- 172.21.15.129:0/3507223282 learned_addr learned my addr 172.21.15.129:0/3507223282 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.790+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5cf805a110 con 0x7f5d040fff20 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.790+0000 7f5cf7fff640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7f5cf00738a0 0x7f5cf0075cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.790+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 --> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cf0076370 con 0x7f5cf00738a0 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.790+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5cf8055cb0 con 0x7f5d040fff20 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.790+0000 7f5d0affd640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7f5cf00738a0 0x7f5cf0075cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.791+0000 7f5d0affd640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7f5cf00738a0 0x7f5cf0075cc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.791+0000 7f8f3d489640 1 Processor -- start 2024-03-23T15:47:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.791+0000 7f8f3d489640 1 -- start start 2024-03-23T15:47:39.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.792+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== osd.0 v2:172.21.15.129:6824/1377293601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cf0076370 con 0x7f5cf00738a0 2024-03-23T15:47:39.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.793+0000 7f8f3d489640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f30046510 0x7f8f30051310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.793+0000 7f8f3d489640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f300d90a0 0x7f8f301794f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.793+0000 7f8f3d489640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f300da540 0x7f8f300d5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.793+0000 7f8f3d489640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f30052130 con 0x7f8f30046510 2024-03-23T15:47:39.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.793+0000 7f8f3d489640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8f30179a30 con 0x7f8f300d90a0 2024-03-23T15:47:39.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.793+0000 7f8f3d489640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f300dbdf0 con 0x7f8f300da540 2024-03-23T15:47:39.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 --> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f050c0ef660 con 0x7f0500073a60 2024-03-23T15:47:39.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f04fb7fe640 1 -- 172.21.15.129:0/1202698842 <== osd.1 v2:172.21.15.129:6816/168045261 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0504001430 con 0x7f0500073a60 2024-03-23T15:47:39.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f0500073a60 msgr2=0x7f0500075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f0513fff640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f0500073a60 0x7f0500075e80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f50e659c640 1 -- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e80c7860 msgr2=0x7f50e81ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f50e659c640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f50e80c7860 0x7f50e81ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f050006b860 msgr2=0x7f050006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f050006b860 0x7f050006dd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f050c04cb20 tx=0x7f05140ac000 comp rx=0 tx=0).stop 2024-03-23T15:47:39.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.795+0000 7f50e659c640 1 -- 172.21.15.129:0/867272251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e814d890 msgr2=0x7f50e81560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f50e659c640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50e814d890 0x7f50e81560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c039f90 msgr2=0x7f050c04f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 --2- 172.21.15.129:0/1202698842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c039f90 0x7f050c04f9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f050800e9e0 tx=0x7f050800eea0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f05137fe640 1 -- 172.21.15.129:0/1202698842 reap_dead start 2024-03-23T15:47:39.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 shutdown_connections 2024-03-23T15:47:39.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 >> 172.21.15.129:0/1202698842 conn(0x7f050c0061a0 msgr2=0x7f050c006580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f50e659c640 1 -- 172.21.15.129:0/867272251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d00094d0 con 0x7f50e80c6b50 2024-03-23T15:47:39.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 shutdown_connections 2024-03-23T15:47:39.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.796+0000 7f0513fff640 1 -- 172.21.15.129:0/1202698842 wait complete. 2024-03-23T15:47:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.800+0000 7f87110c0640 1 -- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f870c1589f0 msgr2=0x7f870c15a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.800+0000 7f87110c0640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f870c1589f0 0x7f870c15a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.800+0000 7f87110c0640 1 -- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870c0724a0 msgr2=0x7f870c136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.800+0000 7f87110c0640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870c0724a0 0x7f870c136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.800+0000 7f87110c0640 1 -- 172.21.15.129:0/3507223282 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc009400 con 0x7f870c0c7f40 2024-03-23T15:47:39.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.804+0000 7f8f3cc88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f300da540 0x7f8f300d5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.804+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75000f040 con 0x7fc7600c68d0 2024-03-23T15:47:39.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.805+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75000d8a0 con 0x7fc7600c68d0 2024-03-23T15:47:39.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.805+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc760072000 con 0x7fc7600c68d0 2024-03-23T15:47:39.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.805+0000 7f50e659c640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e8146f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50d0000c00 tx=0x7f50d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.805+0000 7f87110c0640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c0c7f40 0x7f870c1366c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f870800c450 tx=0x7f870800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.805+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75000dc30 con 0x7fc7600c68d0 2024-03-23T15:47:39.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.805+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7601f01e0 con 0x7fc7600c68d0 2024-03-23T15:47:39.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.806+0000 7f8f3cc88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f300da540 0x7f8f300d5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35788/0 (socket says 172.21.15.129:35788) 2024-03-23T15:47:39.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.806+0000 7f8f3cc88640 1 -- 172.21.15.129:0/3644062307 learned_addr learned my addr 172.21.15.129:0/3644062307 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.807+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc760071e80 con 0x7fc7600c68d0 2024-03-23T15:47:39.810 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:39.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T15:47:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.809+0000 7f8f37fff640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f30046510 0x7f8f30051310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.811+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 --> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d040f8360 con 0x7f5cf00738a0 2024-03-23T15:47:39.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.811+0000 7f5cf7fff640 1 -- 172.21.15.129:0/3580529869 <== osd.0 v2:172.21.15.129:6824/1377293601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5d040f8360 con 0x7f5cf00738a0 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.811+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7f5cf00738a0 msgr2=0x7f5cf0075cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7f5cf00738a0 0x7f5cf0075cc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5cf006b810 msgr2=0x7f5cf006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5cf006b810 0x7f5cf006dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400dd30 tx=0x7f5d0c09b000 comp rx=0 tx=0).stop 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040fff20 msgr2=0x7f5d040f8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 --2- 172.21.15.129:0/3580529869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d040fff20 0x7f5d040f8640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800d830 tx=0x7f5cf800dcf0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d10be0640 1 -- 172.21.15.129:0/3580529869 reap_dead start 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 shutdown_connections 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 >> 172.21.15.129:0/3580529869 conn(0x7f5d04050350 msgr2=0x7f5d040f5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 shutdown_connections 2024-03-23T15:47:39.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.812+0000 7f5d11e42640 1 -- 172.21.15.129:0/3580529869 wait complete. 2024-03-23T15:47:39.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.816+0000 7f8f3cc88640 1 -- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f300d90a0 msgr2=0x7f8f301794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.816+0000 7f8f3cc88640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f300d90a0 0x7f8f301794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.816+0000 7f8f3cc88640 1 -- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f30046510 msgr2=0x7f8f30051310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.816+0000 7f8f3cc88640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f30046510 0x7f8f30051310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.816+0000 7f8f3cc88640 1 -- 172.21.15.129:0/3644062307 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3806a250 con 0x7f8f300da540 2024-03-23T15:47:39.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.819+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9cf800bd10 con 0x7f9d080ca4a0 2024-03-23T15:47:39.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.819+0000 7f9d04ff9640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cf406b890 0x7f9cf406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.819+0000 7f9d077fe640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cf406b890 0x7f9cf406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.820+0000 7f9d077fe640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cf406b890 0x7f9cf406dd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9d080b1b00 tx=0x7f9d000094f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.821+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0003810 con 0x7f50e80c6b50 2024-03-23T15:47:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.821+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d003f070 con 0x7f50e80c6b50 2024-03-23T15:47:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.821+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0003bb0 con 0x7f50e80c6b50 2024-03-23T15:47:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.821+0000 7f50eceed640 1 -- 172.21.15.129:0/867272251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e814eae0 con 0x7f50e80c6b50 2024-03-23T15:47:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.821+0000 7f50eceed640 1 -- 172.21.15.129:0/867272251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e80ccca0 con 0x7f50e80c6b50 2024-03-23T15:47:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080052d0 con 0x7f870c0c7f40 2024-03-23T15:47:39.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.823+0000 7f50eceed640 1 -- 172.21.15.129:0/867272251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f50e80c47b0 con 0x7f50e80c6b50 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9cf805a020 con 0x7f9d080ca4a0 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f9d04ff9640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7f9cf4073a90 0x7f9cf4075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 --> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cf4076560 con 0x7f9cf4073a90 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.823+0000 7f9d0cc4e640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7f9cf4073a90 0x7f9cf4075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8708005430 con 0x7f870c0c7f40 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f8713b25640 1 -- 172.21.15.129:0/3507223282 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870c153b00 con 0x7f870c0c7f40 2024-03-23T15:47:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.822+0000 7f8713b25640 1 -- 172.21.15.129:0/3507223282 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f870c1540c0 con 0x7f870c0c7f40 2024-03-23T15:47:39.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.823+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080052d0 con 0x7f870c0c7f40 2024-03-23T15:47:39.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.823+0000 7f8f3cc88640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f300da540 0x7f8f300d5590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f38069900 tx=0x7f8f38069b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.824+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86e8001030 con 0x7f870c0c7f40 2024-03-23T15:47:39.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.824+0000 7f9d0cc4e640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7f9cf4073a90 0x7f9cf4075eb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.828 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:39.828 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T15:47:39.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.825+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== osd.7 v2:172.21.15.157:6818/2526630177 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cf4076560 con 0x7f9cf4073a90 2024-03-23T15:47:39.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.829+0000 7fe71bfff640 1 -- 172.21.15.129:0/3517955476 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c02ccd0 con 0x7fe72406b780 2024-03-23T15:47:39.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.829+0000 7fe71bfff640 1 -- 172.21.15.129:0/3517955476 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe71c02ce30 con 0x7fe72406b780 2024-03-23T15:47:39.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.829+0000 7fe71bfff640 1 -- 172.21.15.129:0/3517955476 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c034660 con 0x7fe72406b780 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 -- 172.21.15.129:0/3517955476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72406b780 msgr2=0x7fe724073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 --2- 172.21.15.129:0/3517955476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72406b780 0x7fe724073240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009dc0 tx=0x7fe71c02bf20 comp rx=0 tx=0).stop 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 -- 172.21.15.129:0/3517955476 shutdown_connections 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 --2- 172.21.15.129:0/3517955476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72406b780 0x7fe724073240 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 -- 172.21.15.129:0/3517955476 >> 172.21.15.129:0/3517955476 conn(0x7fe724070080 msgr2=0x7fe724070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 -- 172.21.15.129:0/3517955476 shutdown_connections 2024-03-23T15:47:39.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.830+0000 7fe72b39a640 1 -- 172.21.15.129:0/3517955476 wait complete. 2024-03-23T15:47:39.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.832+0000 7fe72b39a640 1 Processor -- start 2024-03-23T15:47:39.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.832+0000 7fe72b39a640 1 -- start start 2024-03-23T15:47:39.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.833+0000 7fe72b39a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe72406b3a0 0x7fe72414cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.833+0000 7fe72b39a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe72406b780 0x7fe7241efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.833+0000 7fe72b39a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72415b0b0 0x7fe72415c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.833+0000 7fe72b39a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe72414d640 con 0x7fe72406b3a0 2024-03-23T15:47:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.833+0000 7fe72b39a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe724157be0 con 0x7fe72415b0b0 2024-03-23T15:47:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.833+0000 7fe72b39a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7241f3230 con 0x7fe72406b780 2024-03-23T15:47:39.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.835+0000 7fe729937640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72415b0b0 0x7fe72415c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.835+0000 7fe729937640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72415b0b0 0x7fe72415c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41242/0 (socket says 172.21.15.129:41242) 2024-03-23T15:47:39.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.835+0000 7fe729937640 1 -- 172.21.15.129:0/1334713114 learned_addr learned my addr 172.21.15.129:0/1334713114 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:39.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.835+0000 7fe729136640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe72406b3a0 0x7fe72414cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.840+0000 7fe729937640 1 -- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe72406b780 msgr2=0x7fe7241efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.840+0000 7fe729937640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe72406b780 0x7fe7241efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.840+0000 7fe729937640 1 -- 172.21.15.129:0/1334713114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe72406b3a0 msgr2=0x7fe72414cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.840+0000 7fe729937640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe72406b3a0 0x7fe72414cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.840+0000 7fe729937640 1 -- 172.21.15.129:0/1334713114 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71c009a70 con 0x7fe72415b0b0 2024-03-23T15:47:39.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.844+0000 7fe729937640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72415b0b0 0x7fe72415c820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe71c02dfd0 tx=0x7fe71c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.848+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c03d600 con 0x7fe72415b0b0 2024-03-23T15:47:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.849+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe71c03d760 con 0x7fe72415b0b0 2024-03-23T15:47:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.849+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c03db10 con 0x7fe72415b0b0 2024-03-23T15:47:39.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.849+0000 7fe72b39a640 1 -- 172.21.15.129:0/1334713114 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72414d850 con 0x7fe72415b0b0 2024-03-23T15:47:39.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.849+0000 7fe72b39a640 1 -- 172.21.15.129:0/1334713114 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7241f37d0 con 0x7fe72415b0b0 2024-03-23T15:47:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.850+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe700001030 con 0x7fe72415b0b0 2024-03-23T15:47:39.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.856+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 --> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d0815a9d0 con 0x7f9cf4073a90 2024-03-23T15:47:39.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.856+0000 7f9d04ff9640 1 -- 172.21.15.129:0/3655483713 <== osd.7 v2:172.21.15.157:6818/2526630177 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9cfc030b20 con 0x7f9cf4073a90 2024-03-23T15:47:39.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.856+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc75000dd90 con 0x7fc7600c68d0 2024-03-23T15:47:39.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7fc75d7fa640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc74006b840 0x7fc74006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7f9cf4073a90 msgr2=0x7f9cf4075eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7f9cf4073a90 0x7f9cf4075eb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cf406b890 msgr2=0x7f9cf406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7fc757fff640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc74006b840 0x7fc74006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cf406b890 0x7f9cf406dd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9d080b1b00 tx=0x7f9d000094f0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d080ca4a0 msgr2=0x7f9d080b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 --2- 172.21.15.129:0/3655483713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d080ca4a0 0x7f9d080b2d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800d800 tx=0x7f9cf800dcc0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0cc4e640 1 -- 172.21.15.129:0/3655483713 reap_dead start 2024-03-23T15:47:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 shutdown_connections 2024-03-23T15:47:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 >> 172.21.15.129:0/3655483713 conn(0x7f9d08061f40 msgr2=0x7f9d080c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.857+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 shutdown_connections 2024-03-23T15:47:39.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.858+0000 7f9d0e6b1640 1 -- 172.21.15.129:0/3655483713 wait complete. 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.858+0000 7fc757fff640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc74006b840 0x7fc74006dd00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc748038000 tx=0x7fc748030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.862+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc750059720 con 0x7fc7600c68d0 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.862+0000 7fc75d7fa640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7fc740074510 0x7fc7400769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.862+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 --> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc740077080 con 0x7fc740074510 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.862+0000 7fc764e98640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7fc740074510 0x7fc7400769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.863+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc750004680 con 0x7fc7600c68d0 2024-03-23T15:47:39.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.863+0000 7fc764e98640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7fc740074510 0x7fc7400769b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.864+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== osd.4 v2:172.21.15.157:6826/4230040520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc740077080 con 0x7fc740074510 2024-03-23T15:47:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.864+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f50d002f050 con 0x7f50e80c6b50 2024-03-23T15:47:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.865+0000 7f50deffd640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f50c806b890 0x7f50c806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.865+0000 7f50dffff640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f50c806b890 0x7f50c806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.866+0000 7f50dffff640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f50c806b890 0x7f50c806dd50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f50d8006a90 tx=0x7f50d8006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.872+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f50d0041070 con 0x7f50e80c6b50 2024-03-23T15:47:39.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.872+0000 7f50deffd640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f50c80745e0 0x7f50c8076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.872+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 --> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50c8077150 con 0x7f50c80745e0 2024-03-23T15:47:39.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.872+0000 7f50e6d9d640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f50c80745e0 0x7f50c8076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.873+0000 7f50e6d9d640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f50c80745e0 0x7f50c8076a80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.874+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== osd.6 v2:172.21.15.157:6810/4270775080 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50c8077150 con 0x7f50c80745e0 2024-03-23T15:47:39.878 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:39.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-23T15:47:39.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.884+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe71c03b040 con 0x7fe72415b0b0 2024-03-23T15:47:39.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.884+0000 7fe719ffb640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe70806b860 0x7fe70806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.885+0000 7fe729136640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe70806b860 0x7fe70806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.885+0000 7fe729136640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe70806b860 0x7fe70806dd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe720000f60 tx=0x7fe720009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.889+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8708012630 con 0x7f870c0c7f40 2024-03-23T15:47:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.890+0000 7f87027fc640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f86f006b870 0x7f86f006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.891+0000 7f87118c1640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f86f006b870 0x7f86f006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.891+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8708059260 con 0x7f870c0c7f40 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.891+0000 7f87027fc640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f86f0073a70 0x7f86f0075e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.891+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 --> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86f0076540 con 0x7f86f0073a70 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.891+0000 7f87120c2640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f86f0073a70 0x7f86f0075e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.892+0000 7f87120c2640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f86f0073a70 0x7f86f0075e90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.894+0000 7f87118c1640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f86f006b870 0x7f86f006dd30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86fc036000 tx=0x7f86fc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.894+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== osd.2 v2:172.21.15.129:6800/2679851475 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f0076540 con 0x7f86f0073a70 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.894+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f380ba830 con 0x7f8f300da540 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.894+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f380ba990 con 0x7f8f300da540 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.894+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f380bad40 con 0x7f8f300da540 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.895+0000 7f8f3d489640 1 -- 172.21.15.129:0/3644062307 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f30051bb0 con 0x7f8f300da540 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.895+0000 7f8f3d489640 1 -- 172.21.15.129:0/3644062307 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f300dc060 con 0x7f8f300da540 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.901+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe71c07bcb0 con 0x7fe72415b0b0 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.901+0000 7fe719ffb640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fe708073a60 0x7fe708075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.901+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 --> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe708076530 con 0x7fe708073a60 2024-03-23T15:47:39.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.901+0000 7fe728935640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fe708073a60 0x7fe708075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:39.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.902+0000 7fe728935640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fe708073a60 0x7fe708075e80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:39.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.903+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== osd.5 v2:172.21.15.157:6802/869595834 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe708076530 con 0x7fe708073a60 2024-03-23T15:47:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.896+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f10001030 con 0x7f8f300da540 2024-03-23T15:47:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.911+0000 7f50eceed640 1 -- 172.21.15.129:0/867272251 --> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f50e80c44d0 con 0x7f50c80745e0 2024-03-23T15:47:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.900+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 --> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc760156300 con 0x7fc740074510 2024-03-23T15:47:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.912+0000 7f50deffd640 1 -- 172.21.15.129:0/867272251 <== osd.6 v2:172.21.15.157:6810/4270775080 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f50e0006b10 con 0x7f50c80745e0 2024-03-23T15:47:39.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.912+0000 7fc75d7fa640 1 -- 172.21.15.129:0/1452281615 <== osd.4 v2:172.21.15.157:6826/4230040520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc7580114a0 con 0x7fc740074510 2024-03-23T15:47:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7fc740074510 msgr2=0x7fc7400769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7fc740074510 0x7fc7400769b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc74006b840 msgr2=0x7fc74006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc74006b840 0x7fc74006dd00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc748038000 tx=0x7fc748030000 comp rx=0 tx=0).stop 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7600c68d0 msgr2=0x7fc7600c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 --2- 172.21.15.129:0/1452281615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7600c68d0 0x7fc7600c9670 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc75000a7a0 tx=0x7fc75000ac60 comp rx=0 tx=0).stop 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc764e98640 1 -- 172.21.15.129:0/1452281615 reap_dead start 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.914+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 shutdown_connections 2024-03-23T15:47:39.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.915+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 >> 172.21.15.129:0/1452281615 conn(0x7fc76006b8f0 msgr2=0x7fc7600c6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.915+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 shutdown_connections 2024-03-23T15:47:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.915+0000 7fc765699640 1 -- 172.21.15.129:0/1452281615 wait complete. 2024-03-23T15:47:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.922+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f50c80745e0 msgr2=0x7f50c8076a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.922+0000 7f50dcff9640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f50c80745e0 0x7f50c8076a80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.922+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f50c806b890 msgr2=0x7f50c806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.922+0000 7f50dcff9640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f50c806b890 0x7f50c806dd50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f50d8006a90 tx=0x7f50d8006a20 comp rx=0 tx=0).stop 2024-03-23T15:47:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.922+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 msgr2=0x7f50e8146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.922+0000 7f50dcff9640 1 --2- 172.21.15.129:0/867272251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f50e80c6b50 0x7f50e8146f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50d0000c00 tx=0x7f50d00046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.926+0000 7f50e6d9d640 1 -- 172.21.15.129:0/867272251 reap_dead start 2024-03-23T15:47:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.928+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 shutdown_connections 2024-03-23T15:47:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.928+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 >> 172.21.15.129:0/867272251 conn(0x7f50e806c240 msgr2=0x7f50e814c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:39.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.928+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 shutdown_connections 2024-03-23T15:47:39.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.928+0000 7f50dcff9640 1 -- 172.21.15.129:0/867272251 wait complete. 2024-03-23T15:47:39.941 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:40.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T15:47:40.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.940+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 --> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f86e8002cb0 con 0x7f86f0073a70 2024-03-23T15:47:40.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.941+0000 7f87027fc640 1 -- 172.21.15.129:0/3507223282 <== osd.2 v2:172.21.15.129:6800/2679851475 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f86e8002cb0 con 0x7f86f0073a70 2024-03-23T15:47:40.049 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:40.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T15:47:40.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f86f0073a70 msgr2=0x7f86f0075e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f86f0073a70 0x7f86f0075e90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f86f006b870 msgr2=0x7f86f006dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f86f006b870 0x7f86f006dd30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86fc036000 tx=0x7f86fc030000 comp rx=0 tx=0).stop 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c0c7f40 msgr2=0x7f870c1366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 --2- 172.21.15.129:0/3507223282 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f870c0c7f40 0x7f870c1366c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f870800c450 tx=0x7f870800c910 comp rx=0 tx=0).stop 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f87120c2640 1 -- 172.21.15.129:0/3507223282 reap_dead start 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 shutdown_connections 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 >> 172.21.15.129:0/3507223282 conn(0x7f870c0cb910 msgr2=0x7f870c0c4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 shutdown_connections 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.950+0000 7f86effff640 1 -- 172.21.15.129:0/3507223282 wait complete. 2024-03-23T15:47:40.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.952+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 --> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe700002d30 con 0x7fe708073a60 2024-03-23T15:47:40.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.952+0000 7fe719ffb640 1 -- 172.21.15.129:0/1334713114 <== osd.5 v2:172.21.15.157:6802/869595834 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe700002d30 con 0x7fe708073a60 2024-03-23T15:47:40.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.953+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fe708073a60 msgr2=0x7fe708075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.953+0000 7fe7077fe640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fe708073a60 0x7fe708075e80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.953+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe70806b860 msgr2=0x7fe70806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.953+0000 7fe7077fe640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe70806b860 0x7fe70806dd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe720000f60 tx=0x7fe720009040 comp rx=0 tx=0).stop 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.953+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72415b0b0 msgr2=0x7fe72415c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.953+0000 7fe7077fe640 1 --2- 172.21.15.129:0/1334713114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe72415b0b0 0x7fe72415c820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe71c02dfd0 tx=0x7fe71c004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.954+0000 7fe729937640 1 -- 172.21.15.129:0/1334713114 reap_dead start 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.954+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 shutdown_connections 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.954+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 >> 172.21.15.129:0/1334713114 conn(0x7fe724070080 msgr2=0x7fe724072f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.954+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 shutdown_connections 2024-03-23T15:47:40.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.954+0000 7fe7077fe640 1 -- 172.21.15.129:0/1334713114 wait complete. 2024-03-23T15:47:40.054 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:40.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T15:47:40.055 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:40.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-23T15:47:40.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.975+0000 7f881a2c0640 1 Processor -- start 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.975+0000 7f881a2c0640 1 -- start start 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.975+0000 7f881a2c0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f88140707a0 0x7f8814150d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.975+0000 7f881a2c0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8814070b80 0x7f8814151270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.976+0000 7f881a2c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88140c4b40 con 0x7f8814158a30 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.976+0000 7f881a2c0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f881415b750 con 0x7f8814070b80 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.976+0000 7f881a2c0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f88140c5d30 con 0x7f88140707a0 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.976+0000 7f8813fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f88140707a0 0x7f8814150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.976+0000 7f8813fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f88140707a0 0x7f8814150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41248/0 (socket says 172.21.15.129:41248) 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.977+0000 7f8813fff640 1 -- 172.21.15.129:0/1034451647 learned_addr learned my addr 172.21.15.129:0/1034451647 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.979+0000 7f8813fff640 1 -- 172.21.15.129:0/1034451647 >> v2:172.21.15.129:3300/0 conn(0x7f8814070b80 msgr2=0x7f8814151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.979+0000 7f8813fff640 1 --2- 172.21.15.129:0/1034451647 >> v2:172.21.15.129:3300/0 conn(0x7f8814070b80 0x7f8814151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.979+0000 7f8813fff640 1 -- 172.21.15.129:0/1034451647 >> v1:172.21.15.157:6789/0 conn(0x7f8814158a30 legacy=0x7f8814158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.979+0000 7f8813fff640 1 -- 172.21.15.129:0/1034451647 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881415a8f0 con 0x7f88140707a0 2024-03-23T15:47:40.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.984+0000 7f8813fff640 1 --2- 172.21.15.129:0/1034451647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88140707a0 0x7f8814150d30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88140704c0 tx=0x7f880402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1b4224a0ba08adc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.987+0000 7f8812ffd640 1 -- 172.21.15.129:0/1034451647 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804030860 con 0x7f88140707a0 2024-03-23T15:47:40.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.986+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8f380ad070 con 0x7f8f300da540 2024-03-23T15:47:40.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.987+0000 7f8f34ff9640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f1c06b890 0x7f8f1c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.987+0000 7f8f37fff640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f1c06b890 0x7f8f1c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.987+0000 7f8f37fff640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f1c06b890 0x7f8f1c06dd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c000f60 tx=0x7f8f2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.989+0000 7f8812ffd640 1 -- 172.21.15.129:0/1034451647 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88040309c0 con 0x7f88140707a0 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.989+0000 7f8812ffd640 1 -- 172.21.15.129:0/1034451647 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804030d50 con 0x7f88140707a0 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 -- 172.21.15.129:0/1034451647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88140707a0 msgr2=0x7f8814150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 --2- 172.21.15.129:0/1034451647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88140707a0 0x7f8814150d30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88140704c0 tx=0x7f880402b9a0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 -- 172.21.15.129:0/1034451647 shutdown_connections 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 --2- 172.21.15.129:0/1034451647 >> v2:172.21.15.129:3300/0 conn(0x7f8814070b80 0x7f8814151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 --2- 172.21.15.129:0/1034451647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88140707a0 0x7f8814150d30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 -- 172.21.15.129:0/1034451647 >> 172.21.15.129:0/1034451647 conn(0x7f881406c240 msgr2=0x7f881406c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 -- 172.21.15.129:0/1034451647 shutdown_connections 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.990+0000 7f881a2c0640 1 -- 172.21.15.129:0/1034451647 wait complete. 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.991+0000 7f881a2c0640 1 Processor -- start 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 -- start start 2024-03-23T15:47:40.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88140707a0 0x7f88141f0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814070b80 0x7f8814072740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 0x7f881414a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f88141ec900 con 0x7f8814070b80 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f88141ecbe0 con 0x7f8814158a30 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.992+0000 7f881a2c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f88141ecec0 con 0x7f88140707a0 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.993+0000 7f881885d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 0x7f881414a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.993+0000 7f881885d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 0x7f881414a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41264/0 (socket says 172.21.15.129:41264) 2024-03-23T15:47:40.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.993+0000 7f881885d640 1 -- 172.21.15.129:0/3109863994 learned_addr learned my addr 172.21.15.129:0/3109863994 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.993+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8f380f6de0 con 0x7f8f300da540 2024-03-23T15:47:40.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.994+0000 7f8f34ff9640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f8f1c073a90 0x7f8f1c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.994+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 --> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f1c076560 con 0x7f8f1c073a90 2024-03-23T15:47:40.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.994+0000 7f8f377fe640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f8f1c073a90 0x7f8f1c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.994+0000 7f8f377fe640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f8f1c073a90 0x7f8f1c075eb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.994+0000 7f8813fff640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88140707a0 0x7f88141f0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.995+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== osd.3 v2:172.21.15.129:6802/3775172849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f1c076560 con 0x7f8f1c073a90 2024-03-23T15:47:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.999+0000 7f881885d640 1 -- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88140707a0 msgr2=0x7f88141f0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.999+0000 7f881885d640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88140707a0 0x7f88141f0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.999+0000 7f881885d640 1 -- 172.21.15.129:0/3109863994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814070b80 msgr2=0x7f8814072740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.999+0000 7f881885d640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814070b80 0x7f8814072740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:39.999+0000 7f881885d640 1 -- 172.21.15.129:0/3109863994 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88040094d0 con 0x7f8814158a30 2024-03-23T15:47:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.002+0000 7f881885d640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 0x7f881414a460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f880c00c730 tx=0x7f880c00cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.003+0000 7f72321af640 1 Processor -- start 2024-03-23T15:47:40.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.003+0000 7f72321af640 1 -- start start 2024-03-23T15:47:40.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.005+0000 7f72321af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 0x7f722c0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.005+0000 7f72321af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f722c06c4f0 0x7f722c0c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.005+0000 7f72321af640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f722c0621b0 con 0x7f722c0c6ab0 2024-03-23T15:47:40.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.005+0000 7f72321af640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f722c061bf0 con 0x7f722c06c4f0 2024-03-23T15:47:40.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.005+0000 7f72321af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f722c061ed0 con 0x7f722c1513b0 2024-03-23T15:47:40.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.005+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 0x7f722c0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.006+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c006d50 con 0x7f8814158a30 2024-03-23T15:47:40.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.006+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88141ed1a0 con 0x7f8814158a30 2024-03-23T15:47:40.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.006+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 0x7f722c0c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35822/0 (socket says 172.21.15.129:35822) 2024-03-23T15:47:40.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.006+0000 7f722b7fe640 1 -- 172.21.15.129:0/2820182309 learned_addr learned my addr 172.21.15.129:0/2820182309 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.006+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881414f0d0 con 0x7f8814158a30 2024-03-23T15:47:40.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.007+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88141ecd20 con 0x7f8814158a30 2024-03-23T15:47:40.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.008+0000 7f722b7fe640 1 -- 172.21.15.129:0/2820182309 >> v2:172.21.15.129:3300/0 conn(0x7f722c06c4f0 msgr2=0x7f722c0c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.008+0000 7f722b7fe640 1 --2- 172.21.15.129:0/2820182309 >> v2:172.21.15.129:3300/0 conn(0x7f722c06c4f0 0x7f722c0c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.008+0000 7f722b7fe640 1 -- 172.21.15.129:0/2820182309 >> v1:172.21.15.129:6789/0 conn(0x7f722c0c6ab0 legacy=0x7f722c0c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.008+0000 7f722b7fe640 1 -- 172.21.15.129:0/2820182309 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722c0c2b50 con 0x7f722c1513b0 2024-03-23T15:47:40.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.011+0000 7f722b7fe640 1 --2- 172.21.15.129:0/2820182309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 0x7f722c0c10d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f721c00bb10 tx=0x7f721c02e170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=691845daccc4b4b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 --> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8f10002d30 con 0x7f8f1c073a90 2024-03-23T15:47:40.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f34ff9640 1 -- 172.21.15.129:0/3644062307 <== osd.3 v2:172.21.15.129:6802/3775172849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8f10002d30 con 0x7f8f1c073a90 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f8f1c073a90 msgr2=0x7f8f1c075eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f8f1c073a90 0x7f8f1c075eb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f1c06b890 msgr2=0x7f8f1c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f1c06b890 0x7f8f1c06dd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c000f60 tx=0x7f8f2c009040 comp rx=0 tx=0).stop 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f300da540 msgr2=0x7f8f300d5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.012+0000 7f8f227fc640 1 --2- 172.21.15.129:0/3644062307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f300da540 0x7f8f300d5590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f38069900 tx=0x7f8f38069b20 comp rx=0 tx=0).stop 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.013+0000 7f8f3cc88640 1 -- 172.21.15.129:0/3644062307 reap_dead start 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.013+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 shutdown_connections 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.013+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 >> 172.21.15.129:0/3644062307 conn(0x7f8f3004dd20 msgr2=0x7f8f300d68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.013+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 shutdown_connections 2024-03-23T15:47:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.013+0000 7f8f227fc640 1 -- 172.21.15.129:0/3644062307 wait complete. 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.016+0000 7f722a7fc640 1 -- 172.21.15.129:0/2820182309 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c036440 con 0x7f722c1513b0 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.016+0000 7f722a7fc640 1 -- 172.21.15.129:0/2820182309 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f721c0365a0 con 0x7f722c1513b0 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.016+0000 7f722a7fc640 1 -- 172.21.15.129:0/2820182309 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c036970 con 0x7f722c1513b0 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 -- 172.21.15.129:0/2820182309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 msgr2=0x7f722c0c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 --2- 172.21.15.129:0/2820182309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 0x7f722c0c10d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f721c00bb10 tx=0x7f721c02e170 comp rx=0 tx=0).stop 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 -- 172.21.15.129:0/2820182309 shutdown_connections 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 --2- 172.21.15.129:0/2820182309 >> v2:172.21.15.129:3300/0 conn(0x7f722c06c4f0 0x7f722c0c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 --2- 172.21.15.129:0/2820182309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c1513b0 0x7f722c0c10d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 -- 172.21.15.129:0/2820182309 >> 172.21.15.129:0/2820182309 conn(0x7f722c0c8bb0 msgr2=0x7f722c0c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 -- 172.21.15.129:0/2820182309 shutdown_connections 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.017+0000 7f72321af640 1 -- 172.21.15.129:0/2820182309 wait complete. 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.018+0000 7f72321af640 1 Processor -- start 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.019+0000 7f72321af640 1 -- start start 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.019+0000 7f72321af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 0x7f722c1ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f72321af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f722c0c6ab0 0x7f722c0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f72321af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f722c1513b0 0x7f722c146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f72321af640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1ddc60 con 0x7f722c0c6ab0 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f72321af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1ef860 con 0x7f722c1513b0 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f72321af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f722c1dd170 con 0x7f722c06c4f0 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 0x7f722c1ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f722b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 0x7f722c1ef320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35830/0 (socket says 172.21.15.129:35830) 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.020+0000 7f722b7fe640 1 -- 172.21.15.129:0/1566518704 learned_addr learned my addr 172.21.15.129:0/1566518704 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.024+0000 7f722b7fe640 1 -- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f722c1513b0 msgr2=0x7f722c146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.024+0000 7f722b7fe640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f722c1513b0 0x7f722c146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.024+0000 7f722b7fe640 1 -- 172.21.15.129:0/1566518704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f722c0c6ab0 msgr2=0x7f722c0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.024+0000 7f722b7fe640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f722c0c6ab0 0x7f722c0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.024+0000 7f722b7fe640 1 -- 172.21.15.129:0/1566518704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721c00b770 con 0x7f722c06c4f0 2024-03-23T15:47:40.077 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:40.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.030+0000 7f722b7fe640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 0x7f722c1ef320 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f721c00b2c0 tx=0x7f721c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.036+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c034b00 con 0x7f722c06c4f0 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.036+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f721c034c80 con 0x7f722c06c4f0 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.036+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722c1de6b0 con 0x7f722c06c4f0 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.036+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c041470 con 0x7f722c06c4f0 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.036+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f722c1dd3c0 con 0x7f722c06c4f0 2024-03-23T15:47:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.037+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f722c0757b0 con 0x7f722c06c4f0 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.060+0000 7f86ae99a640 1 Processor -- start 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.060+0000 7f86ae99a640 1 -- start start 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.061+0000 7f86ae99a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f86a8154590 0x7f86a815d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.061+0000 7f86ae99a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a80ca990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.061+0000 7f86ae99a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a80cc2a0 con 0x7f86a815afb0 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.061+0000 7f86ae99a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f86a815ab20 con 0x7f86a8154590 2024-03-23T15:47:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.061+0000 7f86ae99a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f86a80cbce0 con 0x7f86a80cb780 2024-03-23T15:47:40.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.223+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c003970 con 0x7f8814158a30 2024-03-23T15:47:40.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.224+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c003d20 con 0x7f8814158a30 2024-03-23T15:47:40.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.248+0000 7f73b442e640 1 Processor -- start 2024-03-23T15:47:40.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.248+0000 7f73b442e640 1 -- start start 2024-03-23T15:47:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.251+0000 7f73b442e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f73ac0701f0 0x7f73ac169f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.251+0000 7f73b442e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ac15e810 con 0x7f73ac070c20 2024-03-23T15:47:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.251+0000 7f73b442e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ac073130 con 0x7f73ac158890 2024-03-23T15:47:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.251+0000 7f73b442e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f73ac15e600 con 0x7f73ac0701f0 2024-03-23T15:47:40.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.251+0000 7f73b19c9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f73ac070c20 0x7f73ac06b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36588/0 (socket says 172.21.15.129:36588) 2024-03-23T15:47:40.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.251+0000 7f73b19c9640 1 -- 172.21.15.129:0/336437455 learned_addr learned my addr 172.21.15.129:0/336437455 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:40.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.254+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2018584196 0 0) 0x7f73ac15e810 con 0x7f73ac070c20 2024-03-23T15:47:40.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.255+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7398003520 con 0x7f73ac070c20 2024-03-23T15:47:40.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.260+0000 7f93cbeaf640 1 Processor -- start 2024-03-23T15:47:40.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.260+0000 7f93cbeaf640 1 -- start start 2024-03-23T15:47:40.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f93cbeaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40cd9d0 0x7f93c4161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f93cbeaf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f93c415f260 0x7f93c40cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f93cbeaf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c4162c90 con 0x7f93c40c38a0 2024-03-23T15:47:40.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f93cbeaf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f93c415ee40 con 0x7f93c415f260 2024-03-23T15:47:40.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f93cbeaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93c415f710 con 0x7f93c40cd9d0 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 49000919 0 0) 0x7f7398003520 con 0x7f73ac070c20 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 >> v2:172.21.15.157:3300/0 conn(0x7f73ac0701f0 msgr2=0x7f73ac169f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 --2- 172.21.15.129:0/336437455 >> v2:172.21.15.157:3300/0 conn(0x7f73ac0701f0 0x7f73ac169f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 >> v1:172.21.15.157:6789/0 conn(0x7f73ac158890 legacy=0x7f73ac15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac162600 con 0x7f73ac070c20 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f739c001670 con 0x7f73ac070c20 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 >> v1:172.21.15.129:6789/0 conn(0x7f73ac070c20 legacy=0x7f73ac06b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7398003970 0x7f7398003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73980047c0 0x7f7398004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73980052a0 0x7f739800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac15e600 con 0x7f73980052a0 2024-03-23T15:47:40.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f739800c190 con 0x7f7398003970 2024-03-23T15:47:40.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.261+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f739800c350 con 0x7f73980047c0 2024-03-23T15:47:40.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.263+0000 7f93c944a640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f93c40c38a0 0x7f93c40c87f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36604/0 (socket says 172.21.15.129:36604) 2024-03-23T15:47:40.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.263+0000 7f93c944a640 1 -- 172.21.15.129:0/3528930473 learned_addr learned my addr 172.21.15.129:0/3528930473 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:40.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.266+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397391192 0 0) 0x7f93c0001110 con 0x7f93c40c38a0 2024-03-23T15:47:40.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.266+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93b0003520 con 0x7f93c40c38a0 2024-03-23T15:47:40.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.267+0000 7f73b29cb640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73980052a0 0x7f739800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2965950591 0 0) 0x7f93b0003520 con 0x7f93c40c38a0 2024-03-23T15:47:40.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40cd9d0 msgr2=0x7f93c4161ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40cd9d0 0x7f93c4161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 >> v2:172.21.15.157:3300/0 conn(0x7f93c415f260 msgr2=0x7f93c40cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 --2- 172.21.15.129:0/3528930473 >> v2:172.21.15.157:3300/0 conn(0x7f93c415f260 0x7f93c40cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f14ea0c9640 1 Processor -- start 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c40c3db0 con 0x7f93c40c38a0 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f93c0001800 con 0x7f93c40c38a0 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 >> v1:172.21.15.129:6789/0 conn(0x7f93c40c38a0 legacy=0x7f93c40c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40cd9d0 0x7f93b0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b0004080 0x7f93b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93c415ee40 con 0x7f93b0004d50 2024-03-23T15:47:40.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f93c415f710 con 0x7f93b0004080 2024-03-23T15:47:40.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93b000bcd0 con 0x7f93c40cd9d0 2024-03-23T15:47:40.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.271+0000 7f14ea0c9640 1 -- start start 2024-03-23T15:47:40.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.273+0000 7f14ea0c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14e40c3d80 0x7f14e40c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.273+0000 7f14ea0c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e4156e90 0x7f14e40c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.273+0000 7f14ea0c9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e40c4af0 con 0x7f14e4150dd0 2024-03-23T15:47:40.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.273+0000 7f14ea0c9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f14e40c4250 con 0x7f14e40c3d80 2024-03-23T15:47:40.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.273+0000 7f14ea0c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14e40c4530 con 0x7f14e4156e90 2024-03-23T15:47:40.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.276+0000 7f73b29cb640 1 -- 172.21.15.129:0/336437455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73980047c0 msgr2=0x7f7398004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.276+0000 7f73b29cb640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73980047c0 0x7f7398004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.276+0000 7f73b29cb640 1 -- 172.21.15.129:0/336437455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7398003970 msgr2=0x7f7398003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.276+0000 7f73b29cb640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7398003970 0x7f7398003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.276+0000 7f73b29cb640 1 -- 172.21.15.129:0/336437455 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac162600 con 0x7f73980052a0 2024-03-23T15:47:40.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.277+0000 7fbb028eb640 1 Processor -- start 2024-03-23T15:47:40.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.277+0000 7fbb028eb640 1 -- start start 2024-03-23T15:47:40.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.277+0000 7f73b29cb640 1 -- 172.21.15.129:0/336437455 reap_dead start 2024-03-23T15:47:40.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.278+0000 7fbb028eb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbafc152150 0x7fbafc0c7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.278+0000 7fbb028eb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbafc0724d0 0x7fbafc158c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.278+0000 7fbb028eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc0c9d40 0x7fbafc15b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.278+0000 7fbb028eb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fbafc0c7860 con 0x7fbafc152150 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.278+0000 7fbb028eb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbafc0c7b40 con 0x7fbafc0724d0 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.278+0000 7fbb028eb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbafc0c7e20 con 0x7fbafc0c9d40 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.279+0000 7f95e6e84640 1 Processor -- start 2024-03-23T15:47:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.279+0000 7f95e6e84640 1 -- start start 2024-03-23T15:47:40.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.279+0000 7f95e6e84640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f95e0154590 0x7f95e015d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.279+0000 7f95e6e84640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e00cc2a0 con 0x7f95e015afb0 2024-03-23T15:47:40.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.279+0000 7f95e6e84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e015ab20 con 0x7f95e00cb780 2024-03-23T15:47:40.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.279+0000 7f95e6e84640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f95e00cbce0 con 0x7f95e0154590 2024-03-23T15:47:40.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.281+0000 7f93ca44c640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.281+0000 7f14e37fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f14e4150dd0 0x7f14e40cc7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36616/0 (socket says 172.21.15.129:36616) 2024-03-23T15:47:40.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.281+0000 7f14e37fe640 1 -- 172.21.15.129:0/1382850620 learned_addr learned my addr 172.21.15.129:0/1382850620 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:40.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.281+0000 7f73b29cb640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73980052a0 0x7f739800b980 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009e30 tx=0x7f73980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d9a8846a29997c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.286+0000 7f95e5421640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f95e015afb0 0x7f95e00c6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36632/0 (socket says 172.21.15.129:36632) 2024-03-23T15:47:40.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.286+0000 7f95e5421640 1 -- 172.21.15.129:0/1690896325 learned_addr learned my addr 172.21.15.129:0/1690896325 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:40.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.291+0000 7f93ca44c640 1 -- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40cd9d0 msgr2=0x7f93b0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.291+0000 7f93ca44c640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40cd9d0 0x7f93b0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.291+0000 7f93ca44c640 1 -- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b0004080 msgr2=0x7f93b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.291+0000 7f93ca44c640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b0004080 0x7f93b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.291+0000 7f93ca44c640 1 -- 172.21.15.129:0/3528930473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c40c3db0 con 0x7f93b0004d50 2024-03-23T15:47:40.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.293+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3865885238 0 0) 0x7f14e40c4af0 con 0x7f14e4150dd0 2024-03-23T15:47:40.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.293+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14cc003520 con 0x7f14e4150dd0 2024-03-23T15:47:40.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.294+0000 7f93ca44c640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93b00051d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009e40 tx=0x7f93b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e2658af56b3d39 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.298+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f721c03d070 con 0x7f722c06c4f0 2024-03-23T15:47:40.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.298+0000 7f7213fff640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f721806b840 0x7f721806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.298+0000 7f722affd640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f721806b840 0x7f721806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.299+0000 7f722affd640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f721806b840 0x7f721806dd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f722c0c2320 tx=0x7f7220007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.301+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a402cd70 con 0x7f73980052a0 2024-03-23T15:47:40.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.301+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a4003970 con 0x7f73980052a0 2024-03-23T15:47:40.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.301+0000 7f73b11c8640 1 -- 172.21.15.129:0/336437455 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4003d20 con 0x7f73980052a0 2024-03-23T15:47:40.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 919765573 0 0) 0x7f14cc003520 con 0x7f14e4150dd0 2024-03-23T15:47:40.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 -- 172.21.15.129:0/336437455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73980052a0 msgr2=0x7f739800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73980052a0 0x7f739800b980 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009e30 tx=0x7f73980057f0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e4156e90 msgr2=0x7f14e40c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e4156e90 0x7f14e40c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 >> v2:172.21.15.157:3300/0 conn(0x7f14e40c3d80 msgr2=0x7f14e40c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 --2- 172.21.15.129:0/1382850620 >> v2:172.21.15.157:3300/0 conn(0x7f14e40c3d80 0x7f14e40c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e40c4810 con 0x7f14e4150dd0 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f14d4001670 con 0x7f14e4150dd0 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 >> v1:172.21.15.129:6789/0 conn(0x7f14e4150dd0 legacy=0x7f14e40cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc003970 0x7f14cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e4156e90 0x7f14cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14e40c4250 con 0x7f14cc004d00 2024-03-23T15:47:40.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14e40c4530 con 0x7f14cc003970 2024-03-23T15:47:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14cc010010 con 0x7f14e4156e90 2024-03-23T15:47:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 -- 172.21.15.129:0/336437455 shutdown_connections 2024-03-23T15:47:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 --2- 172.21.15.129:0/336437455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73980052a0 0x7f739800b980 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 -- 172.21.15.129:0/336437455 >> 172.21.15.129:0/336437455 conn(0x7f73ac06e470 msgr2=0x7f73ac06e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 -- 172.21.15.129:0/336437455 shutdown_connections 2024-03-23T15:47:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.302+0000 7f73b442e640 1 -- 172.21.15.129:0/336437455 wait complete. 2024-03-23T15:47:40.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.303+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2171706268 0 0) 0x7f95e00cc2a0 con 0x7f95e015afb0 2024-03-23T15:47:40.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.303+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95d0003520 con 0x7f95e015afb0 2024-03-23T15:47:40.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.304+0000 7f73b442e640 1 Processor -- start 2024-03-23T15:47:40.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.304+0000 7f73b442e640 1 -- start start 2024-03-23T15:47:40.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f721c07c7a0 con 0x7f722c06c4f0 2024-03-23T15:47:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f73b442e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73980052a0 0x7f73ac1e8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f73b442e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 0x7f73ac1e8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f73b442e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73ac151960 0x7f73ac151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f73b442e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1e9f40 con 0x7f73ac155860 2024-03-23T15:47:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f73b442e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1e7dc0 con 0x7f73980052a0 2024-03-23T15:47:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.305+0000 7f73b442e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73ac1ea170 con 0x7f73ac151960 2024-03-23T15:47:40.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.313+0000 7fbafbfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbafc152150 0x7fbafc0c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.314+0000 7fbafbfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbafc152150 0x7fbafc0c7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59048/0 (socket says 172.21.15.129:59048) 2024-03-23T15:47:40.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.314+0000 7fbafbfff640 1 -- 172.21.15.129:0/1441263286 learned_addr learned my addr 172.21.15.129:0/1441263286 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 950145589 0 0) 0x7f95d0003520 con 0x7f95e015afb0 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 >> v2:172.21.15.129:3300/0 conn(0x7f95e0154590 msgr2=0x7f95e015d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 --2- 172.21.15.129:0/1690896325 >> v2:172.21.15.129:3300/0 conn(0x7f95e0154590 0x7f95e015d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 >> v1:172.21.15.157:6789/0 conn(0x7f95e00cb780 legacy=0x7f95e00ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e00cbfc0 con 0x7f95e015afb0 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f95d80019f0 con 0x7f95e015afb0 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 >> v1:172.21.15.129:6789/0 conn(0x7f95e015afb0 legacy=0x7f95e00c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95d0003970 0x7f95d0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d00047c0 0x7f95d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d0005380 0x7f95d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95e00cbce0 con 0x7f95d00047c0 2024-03-23T15:47:40.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95d000c220 con 0x7f95d0003970 2024-03-23T15:47:40.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.315+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95d000c3e0 con 0x7f95d0005380 2024-03-23T15:47:40.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.319+0000 7fbafbfff640 1 -- 172.21.15.129:0/1441263286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc0c9d40 msgr2=0x7fbafc15b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.319+0000 7fbafbfff640 1 --2- 172.21.15.129:0/1441263286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc0c9d40 0x7fbafc15b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.319+0000 7fbafbfff640 1 -- 172.21.15.129:0/1441263286 >> v2:172.21.15.157:3300/0 conn(0x7fbafc0724d0 msgr2=0x7fbafc158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.319+0000 7fbafbfff640 1 --2- 172.21.15.129:0/1441263286 >> v2:172.21.15.157:3300/0 conn(0x7fbafc0724d0 0x7fbafc158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.319+0000 7fbafbfff640 1 -- 172.21.15.129:0/1441263286 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbafc158450 con 0x7fbafc152150 2024-03-23T15:47:40.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.321+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc02cd90 con 0x7f93b0004d50 2024-03-23T15:47:40.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.321+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93bc034440 con 0x7f93b0004d50 2024-03-23T15:47:40.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.321+0000 7f93c8c49640 1 -- 172.21.15.129:0/3528930473 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc034830 con 0x7f93b0004d50 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93cbeaf640 1 -- 172.21.15.129:0/3528930473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 msgr2=0x7f93b00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/3528930473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93b00051d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009e40 tx=0x7f93b00057f0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93ca44c640 1 -- 172.21.15.129:0/3528930473 reap_dead start 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93cbeaf640 1 -- 172.21.15.129:0/3528930473 shutdown_connections 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93cbeaf640 1 -- 172.21.15.129:0/3528930473 >> 172.21.15.129:0/3528930473 conn(0x7f93c406f7d0 msgr2=0x7f93c406fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93cbeaf640 1 -- 172.21.15.129:0/3528930473 shutdown_connections 2024-03-23T15:47:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f93cbeaf640 1 -- 172.21.15.129:0/3528930473 wait complete. 2024-03-23T15:47:40.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.322+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f721c078340 con 0x7f722c06c4f0 2024-03-23T15:47:40.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.323+0000 7fbafbfff640 1 --2- 172.21.15.129:0/1441263286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc152150 0x7fbafc0c7240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbafc15c7a0 tx=0x7fbaec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c1ebc17db5e424 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.323+0000 7f73b19c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 0x7f73ac1e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 Processor -- start 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 -- start start 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93c41f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93c40c53c0 0x7f93c4161290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 0x7f93c40c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41f8820 con 0x7f93b0004d50 2024-03-23T15:47:40.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41f8390 con 0x7f93c40c53c0 2024-03-23T15:47:40.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.324+0000 7f93cbeaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93c40c4e20 con 0x7f93c40c5960 2024-03-23T15:47:40.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.325+0000 7f73b19c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 0x7f73ac1e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59082/0 (socket says 172.21.15.129:59082) 2024-03-23T15:47:40.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.325+0000 7f73b19c9640 1 -- 172.21.15.129:0/3877765743 learned_addr learned my addr 172.21.15.129:0/3877765743 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f880c0050d0 con 0x7f8814158a30 2024-03-23T15:47:40.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f86a77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a80ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f8810ff9640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f880006b810 0x7f880006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f86a77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a80ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35846/0 (socket says 172.21.15.129:35846) 2024-03-23T15:47:40.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f86a77fe640 1 -- 172.21.15.129:0/396100518 learned_addr learned my addr 172.21.15.129:0/396100518 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f8813fff640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f880006b810 0x7f880006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f73b19c9640 1 -- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73ac151960 msgr2=0x7f73ac151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f73b19c9640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73ac151960 0x7f73ac151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f73b19c9640 1 -- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73980052a0 msgr2=0x7f73ac1e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f73b19c9640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73980052a0 0x7f73ac1e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.329+0000 7f73b19c9640 1 -- 172.21.15.129:0/3877765743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac1ea450 con 0x7f73ac155860 2024-03-23T15:47:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.330+0000 7f8813fff640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f880006b810 0x7f880006dcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8804036000 tx=0x7f8804008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.332+0000 7f86a77fe640 1 -- 172.21.15.129:0/396100518 >> v2:172.21.15.157:3300/0 conn(0x7f86a8154590 msgr2=0x7f86a815d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.332+0000 7f86a77fe640 1 --2- 172.21.15.129:0/396100518 >> v2:172.21.15.157:3300/0 conn(0x7f86a8154590 0x7f86a815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.332+0000 7f86a77fe640 1 -- 172.21.15.129:0/396100518 >> v1:172.21.15.157:6789/0 conn(0x7f86a815afb0 legacy=0x7f86a80c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.332+0000 7f86a77fe640 1 -- 172.21.15.129:0/396100518 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a80cbfc0 con 0x7f86a80cb780 2024-03-23T15:47:40.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.333+0000 7f14e3fff640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.334+0000 7f95dffff640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d00047c0 0x7f95d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.335+0000 7f86a77fe640 1 --2- 172.21.15.129:0/396100518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a80ca990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f869c009870 tx=0x7f869c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fcd4f3de0e75cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.338+0000 7fbafaffd640 1 -- 172.21.15.129:0/1441263286 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec030860 con 0x7fbafc152150 2024-03-23T15:47:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.338+0000 7fbafaffd640 1 -- 172.21.15.129:0/1441263286 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec0309c0 con 0x7fbafc152150 2024-03-23T15:47:40.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.338+0000 7fbafaffd640 1 -- 172.21.15.129:0/1441263286 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec030d50 con 0x7fbafc152150 2024-03-23T15:47:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 -- 172.21.15.129:0/1441263286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc152150 msgr2=0x7fbafc0c7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 --2- 172.21.15.129:0/1441263286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc152150 0x7fbafc0c7240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbafc15c7a0 tx=0x7fbaec02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 -- 172.21.15.129:0/1441263286 shutdown_connections 2024-03-23T15:47:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 --2- 172.21.15.129:0/1441263286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc0c9d40 0x7fbafc15b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 --2- 172.21.15.129:0/1441263286 >> v2:172.21.15.157:3300/0 conn(0x7fbafc0724d0 0x7fbafc158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 --2- 172.21.15.129:0/1441263286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc152150 0x7fbafc0c7240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 -- 172.21.15.129:0/1441263286 >> 172.21.15.129:0/1441263286 conn(0x7fbafc06c240 msgr2=0x7fbafc06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 -- 172.21.15.129:0/1441263286 shutdown_connections 2024-03-23T15:47:40.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.339+0000 7fbb028eb640 1 -- 172.21.15.129:0/1441263286 wait complete. 2024-03-23T15:47:40.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.340+0000 7f95e5421640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d0005380 0x7f95d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7f14e3fff640 1 -- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e4156e90 msgr2=0x7f14cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7f14e3fff640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e4156e90 0x7f14cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7f14e3fff640 1 -- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc003970 msgr2=0x7f14cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7f14e3fff640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc003970 0x7f14cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7fbb028eb640 1 Processor -- start 2024-03-23T15:47:40.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7f14e3fff640 1 -- 172.21.15.129:0/1382850620 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e40c4810 con 0x7f14cc004d00 2024-03-23T15:47:40.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.341+0000 7fbb028eb640 1 -- start start 2024-03-23T15:47:40.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7fbb028eb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbafc0724d0 0x7fbafc0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7fbb028eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 0x7fbafc159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7fbb028eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc152150 0x7fbafc15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7fbb028eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc14c3f0 con 0x7fbafc0c9d40 2024-03-23T15:47:40.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7fbb028eb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc14d9e0 con 0x7fbafc0724d0 2024-03-23T15:47:40.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7fbb028eb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbafc14dca0 con 0x7fbafc152150 2024-03-23T15:47:40.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7f95dffff640 1 -- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d0005380 msgr2=0x7f95d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7f95dffff640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d0005380 0x7f95d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7f95dffff640 1 -- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95d0003970 msgr2=0x7f95d0003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:40.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7f95dffff640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95d0003970 0x7f95d0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7f95e5421640 1 -- 172.21.15.129:0/1690896325 reap_dead start 2024-03-23T15:47:40.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.342+0000 7f95dffff640 1 -- 172.21.15.129:0/1690896325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e00cbfc0 con 0x7f95d00047c0 2024-03-23T15:47:40.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.344+0000 7f73b19c9640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 0x7f73ac1e8790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f739c00c7f0 tx=0x7f739c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.345+0000 7f93ca44c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 0x7f93c40c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.347+0000 7f14e3fff640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14cc005180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009e40 tx=0x7f14cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3974376713f27701 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.347+0000 7f95dffff640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d00047c0 0x7f95d0004c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f95cc001650 tx=0x7f95d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761e118407a95d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.348+0000 7f86a6ffd640 1 -- 172.21.15.129:0/396100518 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c0317c0 con 0x7f86a80cb780 2024-03-23T15:47:40.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.348+0000 7f86a6ffd640 1 -- 172.21.15.129:0/396100518 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869c031920 con 0x7f86a80cb780 2024-03-23T15:47:40.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.348+0000 7f86a6ffd640 1 -- 172.21.15.129:0/396100518 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c031cb0 con 0x7f86a80cb780 2024-03-23T15:47:40.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.348+0000 7fbafb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 0x7fbafc159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7fbafb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 0x7fbafc159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59094/0 (socket says 172.21.15.129:59094) 2024-03-23T15:47:40.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7fbafb7fe640 1 -- 172.21.15.129:0/2688268336 learned_addr learned my addr 172.21.15.129:0/2688268336 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f93ca44c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 0x7f93c40c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35914/0 (socket says 172.21.15.129:35914) 2024-03-23T15:47:40.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f93ca44c640 1 -- 172.21.15.129:0/2121224529 learned_addr learned my addr 172.21.15.129:0/2121224529 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 -- 172.21.15.129:0/396100518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 msgr2=0x7f86a80ca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 --2- 172.21.15.129:0/396100518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a80ca990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f869c009870 tx=0x7f869c02b860 comp rx=0 tx=0).stop 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 -- 172.21.15.129:0/396100518 shutdown_connections 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 --2- 172.21.15.129:0/396100518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a80ca990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 --2- 172.21.15.129:0/396100518 >> v2:172.21.15.157:3300/0 conn(0x7f86a8154590 0x7f86a815d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 -- 172.21.15.129:0/396100518 >> 172.21.15.129:0/396100518 conn(0x7f86a8061b50 msgr2=0x7f86a8061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 -- 172.21.15.129:0/396100518 shutdown_connections 2024-03-23T15:47:40.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.349+0000 7f86ae99a640 1 -- 172.21.15.129:0/396100518 wait complete. 2024-03-23T15:47:40.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.351+0000 7f86ae99a640 1 Processor -- start 2024-03-23T15:47:40.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.351+0000 7f86ae99a640 1 -- start start 2024-03-23T15:47:40.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.352+0000 7f86ae99a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a81f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.352+0000 7f86ae99a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86a8154590 0x7f86a81edc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.352+0000 7f86ae99a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a815afb0 0x7f86a80c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.352+0000 7f86ae99a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f86a81f4150 con 0x7f86a815afb0 2024-03-23T15:47:40.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.352+0000 7f86ae99a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f86a8158210 con 0x7f86a8154590 2024-03-23T15:47:40.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.352+0000 7f86ae99a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f86a81595d0 con 0x7f86a80cb780 2024-03-23T15:47:40.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.354+0000 7f93c9c4b640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93c41f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.355+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f880c059750 con 0x7f8814158a30 2024-03-23T15:47:40.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.356+0000 7f93ca44c640 1 -- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93c40c53c0 msgr2=0x7f93c4161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.356+0000 7f93ca44c640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93c40c53c0 0x7f93c4161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.356+0000 7f93ca44c640 1 -- 172.21.15.129:0/2121224529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 msgr2=0x7f93c41f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.356+0000 7f93ca44c640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93c41f4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.356+0000 7f93ca44c640 1 -- 172.21.15.129:0/2121224529 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c40c50e0 con 0x7f93c40c5960 2024-03-23T15:47:40.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.357+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02cd90 con 0x7f14cc004d00 2024-03-23T15:47:40.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.358+0000 7fbb00e88640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc152150 0x7fbafc15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.361+0000 7f93ca44c640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 0x7f93c40c5de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93bc02c800 tx=0x7f93bc004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.361+0000 7fbafb7fe640 1 -- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc152150 msgr2=0x7fbafc15a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.361+0000 7fbafb7fe640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc152150 0x7fbafc15a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.361+0000 7fbafb7fe640 1 -- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbafc0724d0 msgr2=0x7fbafc0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.361+0000 7fbafb7fe640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbafc0724d0 0x7fbafc0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.361+0000 7fbafb7fe640 1 -- 172.21.15.129:0/2688268336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec0094d0 con 0x7fbafc0c9d40 2024-03-23T15:47:40.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.364+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc005270 con 0x7f95d00047c0 2024-03-23T15:47:40.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.366+0000 7fbafb7fe640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 0x7fbafc159de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000b4a0 tx=0x7fbaf000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.366+0000 7f14e27fc640 1 -- 172.21.15.129:0/1382850620 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14dc034440 con 0x7f14cc004d00 2024-03-23T15:47:40.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f86a7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a81f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f880c0552f0 con 0x7f8814158a30 2024-03-23T15:47:40.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1382850620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 msgr2=0x7f14cc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1382850620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14cc005180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009e40 tx=0x7f14cc0057f0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f14e3fff640 1 -- 172.21.15.129:0/1382850620 reap_dead start 2024-03-23T15:47:40.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1382850620 shutdown_connections 2024-03-23T15:47:40.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1382850620 >> 172.21.15.129:0/1382850620 conn(0x7f14e406e470 msgr2=0x7f14e406e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc03f420 con 0x7f93c40c5960 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93bc03f580 con 0x7f93c40c5960 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc03f8f0 con 0x7f93c40c5960 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c41f7a60 con 0x7f93c40c5960 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.367+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c41f8020 con 0x7f93c40c5960 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.368+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1382850620 shutdown_connections 2024-03-23T15:47:40.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.368+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1382850620 wait complete. 2024-03-23T15:47:40.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.368+0000 7f86a7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a81f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35942/0 (socket says 172.21.15.129:35942) 2024-03-23T15:47:40.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.368+0000 7f86a7fff640 1 -- 172.21.15.129:0/2973656764 learned_addr learned my addr 172.21.15.129:0/2973656764 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.369+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c41f81b0 con 0x7f93c40c5960 2024-03-23T15:47:40.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.369+0000 7f86acf37640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a815afb0 0x7f86a80c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.369+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c00cea0 con 0x7f73ac155860 2024-03-23T15:47:40.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.369+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739c0074e0 con 0x7f73ac155860 2024-03-23T15:47:40.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.369+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c004a20 con 0x7f73ac155860 2024-03-23T15:47:40.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.369+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac155210 con 0x7f73ac155860 2024-03-23T15:47:40.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.370+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac154f10 con 0x7f73ac155860 2024-03-23T15:47:40.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.370+0000 7f14ea0c9640 1 Processor -- start 2024-03-23T15:47:40.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.370+0000 7f14ea0c9640 1 -- start start 2024-03-23T15:47:40.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f14ea0c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14e414f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ac1e7f70 con 0x7f73ac155860 2024-03-23T15:47:40.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f14ea0c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 0x7f14e414b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f14ea0c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e414d130 0x7f14e414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f14ea0c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14e414afe0 con 0x7f14cc004d00 2024-03-23T15:47:40.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f14ea0c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14e4156bd0 con 0x7f14e41eb4e0 2024-03-23T15:47:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.371+0000 7f14ea0c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14e41eba90 con 0x7f14e414d130 2024-03-23T15:47:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.373+0000 7f86acf37640 1 -- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 msgr2=0x7f86a81f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.373+0000 7f86acf37640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a81f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.374+0000 7f86acf37640 1 -- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86a8154590 msgr2=0x7f86a81edc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.374+0000 7f86acf37640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86a8154590 0x7f86a81edc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.374+0000 7f86acf37640 1 -- 172.21.15.129:0/2973656764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869c0094d0 con 0x7f86a815afb0 2024-03-23T15:47:40.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.385+0000 7f86acf37640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a815afb0 0x7f86a80c1e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f86a000be20 tx=0x7f86a000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.394+0000 7f14e2ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 0x7f14e414b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.396+0000 7f14e2ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 0x7f14e414b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41380/0 (socket says 172.21.15.129:41380) 2024-03-23T15:47:40.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.396+0000 7f14e2ffd640 1 -- 172.21.15.129:0/1635289085 learned_addr learned my addr 172.21.15.129:0/1635289085 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.407+0000 7f14e2ffd640 1 -- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e414d130 msgr2=0x7f14e414b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.407+0000 7f14e2ffd640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e414d130 0x7f14e414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.407+0000 7f14e2ffd640 1 -- 172.21.15.129:0/1635289085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 msgr2=0x7f14e414f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.407+0000 7f14e2ffd640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14e414f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.407+0000 7f14e2ffd640 1 -- 172.21.15.129:0/1635289085 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e41ebd70 con 0x7f14e41eb4e0 2024-03-23T15:47:40.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.412+0000 7f14e2ffd640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 0x7f14e414b470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f14d800b340 tx=0x7f14d800b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.418+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8011830 con 0x7f14e41eb4e0 2024-03-23T15:47:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.418+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d8011990 con 0x7f14e41eb4e0 2024-03-23T15:47:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.418+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8011d40 con 0x7f14e41eb4e0 2024-03-23T15:47:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.418+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e41ec050 con 0x7f14e41eb4e0 2024-03-23T15:47:40.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.418+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e414ac70 con 0x7f14e41eb4e0 2024-03-23T15:47:40.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.419+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e414ae30 con 0x7f14e41eb4e0 2024-03-23T15:47:40.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.448+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc02e070 con 0x7f95d00047c0 2024-03-23T15:47:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.448+0000 7f95df7fe640 1 -- 172.21.15.129:0/1690896325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc003610 con 0x7f95d00047c0 2024-03-23T15:47:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.449+0000 7f95e6e84640 1 -- 172.21.15.129:0/1690896325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d00047c0 msgr2=0x7f95d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.449+0000 7f95e6e84640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d00047c0 0x7f95d0004c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f95cc001650 tx=0x7f95d00057f0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.450+0000 7f95e6e84640 1 -- 172.21.15.129:0/1690896325 shutdown_connections 2024-03-23T15:47:40.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.450+0000 7f95e6e84640 1 --2- 172.21.15.129:0/1690896325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95d00047c0 0x7f95d0004c40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.450+0000 7f95e6e84640 1 -- 172.21.15.129:0/1690896325 >> 172.21.15.129:0/1690896325 conn(0x7f95e0061b50 msgr2=0x7f95e0061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.450+0000 7f95e6e84640 1 -- 172.21.15.129:0/1690896325 shutdown_connections 2024-03-23T15:47:40.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.450+0000 7f95e6e84640 1 -- 172.21.15.129:0/1690896325 wait complete. 2024-03-23T15:47:40.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.452+0000 7f95e6e84640 1 Processor -- start 2024-03-23T15:47:40.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.452+0000 7f95e6e84640 1 -- start start 2024-03-23T15:47:40.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.453+0000 7f95e6e84640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 0x7f95e0150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.454+0000 7f95e6e84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95e0151930 0x7f95e0157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.454+0000 7f95e6e84640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e0154d10 0x7f95e0155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.454+0000 7f95e6e84640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0156520 con 0x7f95e0151930 2024-03-23T15:47:40.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.454+0000 7f95e6e84640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95e014fbd0 con 0x7f95e0154d10 2024-03-23T15:47:40.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.454+0000 7f95e6e84640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95e01556d0 con 0x7f95d00047c0 2024-03-23T15:47:40.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.583+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f88141ecd20 con 0x7f8814158a30 2024-03-23T15:47:40.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.648+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f722c154a00 con 0x7f722c06c4f0 2024-03-23T15:47:40.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.671+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f93bc03fa50 con 0x7f93c40c5960 2024-03-23T15:47:40.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.672+0000 7f93ba7fc640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f93a806b860 0x7f93a806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.672+0000 7f93c9c4b640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f93a806b860 0x7f93a806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.673+0000 7f93c9c4b640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f93a806b860 0x7f93a806dd20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f93b400a7b0 tx=0x7f93b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.676+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f93bc07b960 con 0x7f93c40c5960 2024-03-23T15:47:40.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.686+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f93bc077c00 con 0x7f93c40c5960 2024-03-23T15:47:40.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.687+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f14d801a400 con 0x7f14e41eb4e0 2024-03-23T15:47:40.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.687+0000 7f14cbfff640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f14d006b890 0x7f14d006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.687+0000 7f14e37fe640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f14d006b890 0x7f14d006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.688+0000 7f14e37fe640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f14d006b890 0x7f14d006dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14d4007ae0 tx=0x7f14d4001660 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.689+0000 7f95e4c20640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 0x7f95e0150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.691+0000 7f95e4c20640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 0x7f95e0150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35970/0 (socket says 172.21.15.129:35970) 2024-03-23T15:47:40.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.691+0000 7f95e4c20640 1 -- 172.21.15.129:0/2924653947 learned_addr learned my addr 172.21.15.129:0/2924653947 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:40.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.692+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f14d8059060 con 0x7f14e41eb4e0 2024-03-23T15:47:40.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.702+0000 7f7213fff640 1 -- 172.21.15.129:0/1566518704 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f721c03b020 con 0x7f722c06c4f0 2024-03-23T15:47:40.705 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.704+0000 7f95e5421640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e0154d10 0x7f95e0155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f721806b840 msgr2=0x7f721806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f721806b840 0x7f721806dd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f722c0c2320 tx=0x7f7220007990 comp rx=0 tx=0).stop 2024-03-23T15:47:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 msgr2=0x7f722c1ef320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 0x7f722c1ef320 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f721c00b2c0 tx=0x7f721c0046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 shutdown_connections 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f721806b840 0x7f721806dd00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f722c1513b0 0x7f722c146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f722c0c6ab0 0x7f722c0cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 --2- 172.21.15.129:0/1566518704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f722c06c4f0 0x7f722c1ef320 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 >> 172.21.15.129:0/1566518704 conn(0x7f722c0c8bb0 msgr2=0x7f722c14cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 shutdown_connections 2024-03-23T15:47:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.705+0000 7f72321af640 1 -- 172.21.15.129:0/1566518704 wait complete. 2024-03-23T15:47:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.706+0000 7f95e4c20640 1 -- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e0154d10 msgr2=0x7f95e0155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.706+0000 7f95e4c20640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e0154d10 0x7f95e0155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.706+0000 7f95e4c20640 1 -- 172.21.15.129:0/2924653947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95e0151930 msgr2=0x7f95e0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.706+0000 7f95e4c20640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95e0151930 0x7f95e0157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.706+0000 7f95e4c20640 1 -- 172.21.15.129:0/2924653947 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e0151760 con 0x7f95d00047c0 2024-03-23T15:47:40.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.708+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f14d8055300 con 0x7f14e41eb4e0 2024-03-23T15:47:40.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.711+0000 7f8810ff9640 1 -- 172.21.15.129:0/3109863994 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f88141ecd20 con 0x7f8814158a30 2024-03-23T15:47:40.714 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:40.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.714+0000 7f95e4c20640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 0x7f95e0150f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f95d400b430 tx=0x7f95d400b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.714+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f880006b810 msgr2=0x7f880006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.714+0000 7f881a2c0640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f880006b810 0x7f880006dcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8804036000 tx=0x7f8804008030 comp rx=0 tx=0).stop 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.714+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 msgr2=0x7f881414a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.714+0000 7f881a2c0640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 0x7f881414a460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f880c00c730 tx=0x7f880c00cb30 comp rx=0 tx=0).stop 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 shutdown_connections 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f880006b810 0x7f880006dcd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8814158a30 0x7f881414a460 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814070b80 0x7f8814072740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 --2- 172.21.15.129:0/3109863994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88140707a0 0x7f88141f0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 >> 172.21.15.129:0/3109863994 conn(0x7f881406c240 msgr2=0x7f88140c7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 shutdown_connections 2024-03-23T15:47:40.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.715+0000 7f881a2c0640 1 -- 172.21.15.129:0/3109863994 wait complete. 2024-03-23T15:47:40.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-03-23T15:47:40.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.718+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4011830 con 0x7f95d00047c0 2024-03-23T15:47:40.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.719+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d4011990 con 0x7f95d00047c0 2024-03-23T15:47:40.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.719+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4011d40 con 0x7f95d00047c0 2024-03-23T15:47:40.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.719+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e01efb70 con 0x7f95d00047c0 2024-03-23T15:47:40.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.719+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e01f0130 con 0x7f95d00047c0 2024-03-23T15:47:40.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.720+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e0068c40 con 0x7f95d00047c0 2024-03-23T15:47:40.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2024-03-23T15:47:40.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.766+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0010610 con 0x7fbafc0c9d40 2024-03-23T15:47:40.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.767+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf0010770 con 0x7fbafc0c9d40 2024-03-23T15:47:40.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.767+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0010b20 con 0x7fbafc0c9d40 2024-03-23T15:47:40.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.767+0000 7fbb028eb640 1 -- 172.21.15.129:0/2688268336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbafc14df80 con 0x7fbafc0c9d40 2024-03-23T15:47:40.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.767+0000 7fbb028eb640 1 -- 172.21.15.129:0/2688268336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbafc14c600 con 0x7fbafc0c9d40 2024-03-23T15:47:40.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.768+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad4005290 con 0x7fbafc0c9d40 2024-03-23T15:47:40.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.807+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f739c004b80 con 0x7f73ac155860 2024-03-23T15:47:40.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.808+0000 7f73a2ffd640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f739006b860 0x7f739006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.808+0000 7f73b21ca640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f739006b860 0x7f739006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.809+0000 7f73b21ca640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f739006b860 0x7f739006dd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f73ac050c50 tx=0x7f73a8007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.863+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f739c007650 con 0x7f73ac155860 2024-03-23T15:47:40.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.863+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f739c054860 con 0x7f73ac155860 2024-03-23T15:47:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.872+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a000ce70 con 0x7f86a815afb0 2024-03-23T15:47:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.872+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86a0014440 con 0x7f86a815afb0 2024-03-23T15:47:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.872+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a81598b0 con 0x7f86a815afb0 2024-03-23T15:47:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.872+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a81f0230 con 0x7f86a815afb0 2024-03-23T15:47:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.872+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a0014830 con 0x7f86a815afb0 2024-03-23T15:47:40.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.875+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a81f0750 con 0x7f86a815afb0 2024-03-23T15:47:40.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.941+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f86a001d3b0 con 0x7f86a815afb0 2024-03-23T15:47:40.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.942+0000 7f86a4ff9640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f869406b860 0x7f869406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.942+0000 7f86a7fff640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f869406b860 0x7f869406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.943+0000 7f86a7fff640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f869406b860 0x7f869406dd20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8698009950 tx=0x7f8698008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.948+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f86a0059570 con 0x7f86a815afb0 2024-03-23T15:47:40.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.956+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f14e414b150 con 0x7f14e41eb4e0 2024-03-23T15:47:40.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.960+0000 7f14cbfff640 1 -- 172.21.15.129:0/1635289085 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f14d8015020 con 0x7f14e41eb4e0 2024-03-23T15:47:40.962 INFO:teuthology.orchestra.run.smithi129.stdout:55834574855 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f14d006b890 msgr2=0x7f14d006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f14d006b890 0x7f14d006dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14d4007ae0 tx=0x7f14d4001660 comp rx=0 tx=0).stop 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 msgr2=0x7f14e414b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 0x7f14e414b470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f14d800b340 tx=0x7f14d800b800 comp rx=0 tx=0).stop 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 shutdown_connections 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14e414d130 0x7f14e414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f14d006b890 0x7f14d006dd50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14e41eb4e0 0x7f14e414b470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 --2- 172.21.15.129:0/1635289085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc004d00 0x7f14e414f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:40.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 >> 172.21.15.129:0/1635289085 conn(0x7f14e406e470 msgr2=0x7f14e406e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:40.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 shutdown_connections 2024-03-23T15:47:40.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.962+0000 7f14ea0c9640 1 -- 172.21.15.129:0/1635289085 wait complete. 2024-03-23T15:47:40.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.972+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbaf0010c80 con 0x7fbafc0c9d40 2024-03-23T15:47:40.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.972+0000 7fbaf8ff9640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbae806b890 0x7fbae806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.973+0000 7fbafbfff640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbae806b890 0x7fbae806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.974+0000 7fbafbfff640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbae806b890 0x7fbae806dd50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbaec036000 tx=0x7fbaec03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:40.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-03-23T15:47:40.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.992+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f95d4023430 con 0x7f95d00047c0 2024-03-23T15:47:40.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.992+0000 7f95dd7fa640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95c406b860 0x7f95c406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:40.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.993+0000 7f95dffff640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95c406b860 0x7f95c406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:40.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.993+0000 7f95dffff640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95c406b860 0x7f95c406dd20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f95cc033000 tx=0x7f95cc005000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:40.999+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f95d40592e0 con 0x7f95d00047c0 2024-03-23T15:47:41.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.010+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f95d4054e80 con 0x7f95d00047c0 2024-03-23T15:47:41.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.061+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f93c415e080 con 0x7f93c40c5960 2024-03-23T15:47:41.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.062+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f73ac1e7f30 con 0x7f73ac155860 2024-03-23T15:47:41.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.066+0000 7f93ba7fc640 1 -- 172.21.15.129:0/2121224529 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f93bc039020 con 0x7f93c40c5960 2024-03-23T15:47:41.069 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f93a806b860 msgr2=0x7f93a806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f93a806b860 0x7f93a806dd20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f93b400a7b0 tx=0x7f93b4008040 comp rx=0 tx=0).stop 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 msgr2=0x7f93c40c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 0x7f93c40c5de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93bc02c800 tx=0x7f93bc004b50 comp rx=0 tx=0).stop 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 shutdown_connections 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93c40c5960 0x7f93c40c5de0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f93a806b860 0x7f93a806dd20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93c40c53c0 0x7f93c4161290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 --2- 172.21.15.129:0/2121224529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b0004d50 0x7f93c41f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 >> 172.21.15.129:0/2121224529 conn(0x7f93c406f7d0 msgr2=0x7f93c406c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 shutdown_connections 2024-03-23T15:47:41.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.069+0000 7f93cbeaf640 1 -- 172.21.15.129:0/2121224529 wait complete. 2024-03-23T15:47:41.083 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2024-03-23T15:47:41.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.243+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f95e00757b0 con 0x7f95d00047c0 2024-03-23T15:47:41.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.248+0000 7f95dd7fa640 1 -- 172.21.15.129:0/2924653947 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f95d4015020 con 0x7f95d00047c0 2024-03-23T15:47:41.251 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:41.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95c406b860 msgr2=0x7f95c406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95c406b860 0x7f95c406dd20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f95cc033000 tx=0x7f95cc005000 comp rx=0 tx=0).stop 2024-03-23T15:47:41.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 msgr2=0x7f95e0150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 0x7f95e0150f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f95d400b430 tx=0x7f95d400b8f0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 shutdown_connections 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95c406b860 0x7f95c406dd20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e0154d10 0x7f95e0155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95e0151930 0x7f95e0157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 --2- 172.21.15.129:0/2924653947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95d00047c0 0x7f95e0150f40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 >> 172.21.15.129:0/2924653947 conn(0x7f95e0061b50 msgr2=0x7f95e00ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.250+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 shutdown_connections 2024-03-23T15:47:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.251+0000 7f95e6e84640 1 -- 172.21.15.129:0/2924653947 wait complete. 2024-03-23T15:47:41.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-03-23T15:47:41.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.459+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f86a0055110 con 0x7f86a815afb0 2024-03-23T15:47:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.485+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbaf00590b0 con 0x7fbafc0c9d40 2024-03-23T15:47:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.485+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbaf0054c50 con 0x7fbafc0c9d40 2024-03-23T15:47:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.544+0000 7f73a2ffd640 1 -- 172.21.15.129:0/3877765743 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f739c018020 con 0x7f73ac155860 2024-03-23T15:47:41.547 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.547+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f739006b860 msgr2=0x7f739006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.547+0000 7f73b442e640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f739006b860 0x7f739006dd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f73ac050c50 tx=0x7f73a8007990 comp rx=0 tx=0).stop 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 msgr2=0x7f73ac1e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 0x7f73ac1e8790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f739c00c7f0 tx=0x7f739c00ccb0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 shutdown_connections 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73ac151960 0x7f73ac151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73ac155860 0x7f73ac1e8790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f739006b860 0x7f739006dd20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 --2- 172.21.15.129:0/3877765743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73980052a0 0x7f73ac1e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 >> 172.21.15.129:0/3877765743 conn(0x7f73ac06e470 msgr2=0x7f73ac0c59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 shutdown_connections 2024-03-23T15:47:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.548+0000 7f73b442e640 1 -- 172.21.15.129:0/3877765743 wait complete. 2024-03-23T15:47:41.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-03-23T15:47:41.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.681+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbad4005810 con 0x7fbafc0c9d40 2024-03-23T15:47:41.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.706+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f86a81f0750 con 0x7f86a815afb0 2024-03-23T15:47:41.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T15:47:41.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T15:47:41.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.769+0000 7f86a4ff9640 1 -- 172.21.15.129:0/2973656764 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f86a0019020 con 0x7f86a815afb0 2024-03-23T15:47:41.772 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:41.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f869406b860 msgr2=0x7f869406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f869406b860 0x7f869406dd20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8698009950 tx=0x7f8698008040 comp rx=0 tx=0).stop 2024-03-23T15:47:41.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a815afb0 msgr2=0x7f86a80c1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a815afb0 0x7f86a80c1e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f86a000be20 tx=0x7f86a000bf10 comp rx=0 tx=0).stop 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 shutdown_connections 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86a815afb0 0x7f86a80c1e10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f869406b860 0x7f869406dd20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86a8154590 0x7f86a81edc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 --2- 172.21.15.129:0/2973656764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86a80cb780 0x7f86a81f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 >> 172.21.15.129:0/2973656764 conn(0x7f86a8061b50 msgr2=0x7f86a8153500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 shutdown_connections 2024-03-23T15:47:41.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.773+0000 7f86ae99a640 1 -- 172.21.15.129:0/2973656764 wait complete. 2024-03-23T15:47:41.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.776+0000 7fbaf8ff9640 1 -- 172.21.15.129:0/2688268336 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbaf001e070 con 0x7fbafc0c9d40 2024-03-23T15:47:41.779 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:41.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.780+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbae806b890 msgr2=0x7fbae806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.780+0000 7fbae27fc640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbae806b890 0x7fbae806dd50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbaec036000 tx=0x7fbaec03b040 comp rx=0 tx=0).stop 2024-03-23T15:47:41.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.780+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 msgr2=0x7fbafc159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.780+0000 7fbae27fc640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 0x7fbafc159de0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000b4a0 tx=0x7fbaf000b960 comp rx=0 tx=0).stop 2024-03-23T15:47:41.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.781+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 shutdown_connections 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.781+0000 7fbae27fc640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbafc152150 0x7fbafc15a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.781+0000 7fbae27fc640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbafc0c9d40 0x7fbafc159de0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.781+0000 7fbae27fc640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbae806b890 0x7fbae806dd50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.781+0000 7fbae27fc640 1 --2- 172.21.15.129:0/2688268336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbafc0724d0 0x7fbafc0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.781+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 >> 172.21.15.129:0/2688268336 conn(0x7fbafc06c240 msgr2=0x7fbafc06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.782+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 shutdown_connections 2024-03-23T15:47:41.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.782+0000 7fbae27fc640 1 -- 172.21.15.129:0/2688268336 wait complete. 2024-03-23T15:47:41.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-03-23T15:47:41.794 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:41.799 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-03-23T15:47:41.799 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:41.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.832+0000 7f350f59e640 1 Processor -- start 2024-03-23T15:47:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.832+0000 7f350f59e640 1 -- start start 2024-03-23T15:47:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.833+0000 7f350f59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351006cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.833+0000 7f350f59e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3510070290 0x7f351006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.833+0000 7f350f59e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f351015e690 con 0x7f351006fe50 2024-03-23T15:47:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.833+0000 7f350f59e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f351015e4a0 con 0x7f3510070290 2024-03-23T15:47:41.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.833+0000 7f350f59e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f351015e970 con 0x7f3510158750 2024-03-23T15:47:41.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.833+0000 7f350ed9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3510070290 0x7f351006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.834+0000 7f350dd9b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f351006fe50 0x7f35100c9f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44666/0 (socket says 172.21.15.129:44666) 2024-03-23T15:47:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.834+0000 7f350dd9b640 1 -- 172.21.15.129:0/206156096 learned_addr learned my addr 172.21.15.129:0/206156096 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:41.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.835+0000 7f350e59c640 1 --2- 172.21.15.129:0/206156096 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351006cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.838+0000 7f350ed9d640 1 -- 172.21.15.129:0/206156096 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 msgr2=0x7f351006cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:41.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.838+0000 7f350ed9d640 1 --2- 172.21.15.129:0/206156096 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351006cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.838+0000 7f350ed9d640 1 -- 172.21.15.129:0/206156096 >> v1:172.21.15.129:6789/0 conn(0x7f351006fe50 legacy=0x7f35100c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.838+0000 7f350ed9d640 1 -- 172.21.15.129:0/206156096 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351015ec50 con 0x7f3510070290 2024-03-23T15:47:41.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.841+0000 7f350ed9d640 1 --2- 172.21.15.129:0/206156096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351006ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3508009d90 tx=0x7f350802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2254c68b4e1b2bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.848+0000 7f350d59a640 1 -- 172.21.15.129:0/206156096 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350802cc60 con 0x7f3510070290 2024-03-23T15:47:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.848+0000 7f350d59a640 1 -- 172.21.15.129:0/206156096 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f350802cdc0 con 0x7f3510070290 2024-03-23T15:47:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.848+0000 7f350d59a640 1 -- 172.21.15.129:0/206156096 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508034660 con 0x7f3510070290 2024-03-23T15:47:41.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 -- 172.21.15.129:0/206156096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 msgr2=0x7f351006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 --2- 172.21.15.129:0/206156096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351006ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3508009d90 tx=0x7f350802beb0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 -- 172.21.15.129:0/206156096 shutdown_connections 2024-03-23T15:47:41.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 --2- 172.21.15.129:0/206156096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351006ee00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 --2- 172.21.15.129:0/206156096 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351006cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 -- 172.21.15.129:0/206156096 >> 172.21.15.129:0/206156096 conn(0x7f35100622c0 msgr2=0x7f3510070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 -- 172.21.15.129:0/206156096 shutdown_connections 2024-03-23T15:47:41.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.849+0000 7f350f59e640 1 -- 172.21.15.129:0/206156096 wait complete. 2024-03-23T15:47:41.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 Processor -- start 2024-03-23T15:47:41.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 -- start start 2024-03-23T15:47:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f351006fe50 0x7f35101f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351014cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351014d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f351014cb00 con 0x7f3510070290 2024-03-23T15:47:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f35101f4ca0 con 0x7f351006fe50 2024-03-23T15:47:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.851+0000 7f350f59e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35100c7930 con 0x7f3510158750 2024-03-23T15:47:41.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.852+0000 7f350ed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351014d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.852+0000 7f350dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351014cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.852+0000 7f350e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f351006fe50 0x7f35101f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.852+0000 7f350ed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351014d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48826/0 (socket says 172.21.15.129:48826) 2024-03-23T15:47:41.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.852+0000 7f350ed9d640 1 -- 172.21.15.129:0/3355493863 learned_addr learned my addr 172.21.15.129:0/3355493863 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:41.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.856+0000 7f350dd9b640 1 -- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 msgr2=0x7f351014d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.856+0000 7f350dd9b640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351014d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.856+0000 7f350dd9b640 1 -- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f351006fe50 msgr2=0x7f35101f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.856+0000 7f350dd9b640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f351006fe50 0x7f35101f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.856+0000 7f350dd9b640 1 -- 172.21.15.129:0/3355493863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3508009a70 con 0x7f3510070290 2024-03-23T15:47:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.859+0000 7f350dd9b640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351014cf20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3504000f60 tx=0x7f350400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.864+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504005ce0 con 0x7f3510070290 2024-03-23T15:47:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.864+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3504005e40 con 0x7f3510070290 2024-03-23T15:47:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.864+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504007310 con 0x7f3510070290 2024-03-23T15:47:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.864+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35100c6df0 con 0x7f3510070290 2024-03-23T15:47:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.864+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35100c7370 con 0x7f3510070290 2024-03-23T15:47:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.866+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f351014cc40 con 0x7f3510070290 2024-03-23T15:47:41.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.882+0000 7ff6af363640 1 Processor -- start 2024-03-23T15:47:41.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.882+0000 7ff6af363640 1 -- start start 2024-03-23T15:47:41.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.883+0000 7ff6af363640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a815d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.883+0000 7ff6af363640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff6a80cb780 0x7ff6a80ca990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.883+0000 7ff6af363640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a80cc2a0 con 0x7ff6a815afb0 2024-03-23T15:47:41.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.883+0000 7ff6af363640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a815ab20 con 0x7ff6a80cb780 2024-03-23T15:47:41.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.883+0000 7ff6af363640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a80cbce0 con 0x7ff6a8154590 2024-03-23T15:47:41.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.883+0000 7ff6ad0ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.884+0000 7ff6ad0ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a815d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48832/0 (socket says 172.21.15.129:48832) 2024-03-23T15:47:41.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.884+0000 7ff6ad0ff640 1 -- 172.21.15.129:0/1891163764 learned_addr learned my addr 172.21.15.129:0/1891163764 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:41.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.885+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3504007470 con 0x7f3510070290 2024-03-23T15:47:41.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.885+0000 7f34feffd640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f34f006b890 0x7f34f006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.885+0000 7f350e59c640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f34f006b890 0x7f34f006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.886+0000 7f350e59c640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f34f006b890 0x7f34f006dd50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3500009aa0 tx=0x7f3500009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.887+0000 7ff6ad0ff640 1 -- 172.21.15.129:0/1891163764 >> v2:172.21.15.129:3300/0 conn(0x7ff6a80cb780 msgr2=0x7ff6a80ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:41.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.887+0000 7ff6ad0ff640 1 --2- 172.21.15.129:0/1891163764 >> v2:172.21.15.129:3300/0 conn(0x7ff6a80cb780 0x7ff6a80ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.887+0000 7ff6ad0ff640 1 -- 172.21.15.129:0/1891163764 >> v1:172.21.15.129:6789/0 conn(0x7ff6a815afb0 legacy=0x7ff6a80c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:41.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.887+0000 7ff6ad0ff640 1 -- 172.21.15.129:0/1891163764 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a80cbfc0 con 0x7ff6a8154590 2024-03-23T15:47:41.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.888+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3504007720 con 0x7f3510070290 2024-03-23T15:47:41.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.890+0000 7ff6ad0ff640 1 --2- 172.21.15.129:0/1891163764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a815d920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009870 tx=0x7ff6a402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a630b504f6de0ad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.895+0000 7ff69ffff640 1 -- 172.21.15.129:0/1891163764 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40317c0 con 0x7ff6a8154590 2024-03-23T15:47:41.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.895+0000 7ff69ffff640 1 -- 172.21.15.129:0/1891163764 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a4031920 con 0x7ff6a8154590 2024-03-23T15:47:41.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.895+0000 7ff69ffff640 1 -- 172.21.15.129:0/1891163764 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4031cb0 con 0x7ff6a8154590 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 -- 172.21.15.129:0/1891163764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 msgr2=0x7ff6a815d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 --2- 172.21.15.129:0/1891163764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a815d920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009870 tx=0x7ff6a402b860 comp rx=0 tx=0).stop 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 -- 172.21.15.129:0/1891163764 shutdown_connections 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 --2- 172.21.15.129:0/1891163764 >> v2:172.21.15.129:3300/0 conn(0x7ff6a80cb780 0x7ff6a80ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 --2- 172.21.15.129:0/1891163764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a815d920 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 -- 172.21.15.129:0/1891163764 >> 172.21.15.129:0/1891163764 conn(0x7ff6a8061b50 msgr2=0x7ff6a8061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 -- 172.21.15.129:0/1891163764 shutdown_connections 2024-03-23T15:47:41.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.896+0000 7ff6af363640 1 -- 172.21.15.129:0/1891163764 wait complete. 2024-03-23T15:47:41.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.898+0000 7ff6af363640 1 Processor -- start 2024-03-23T15:47:41.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.898+0000 7ff6af363640 1 -- start start 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.899+0000 7ff6af363640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a80cb780 0x7ff6a814e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.899+0000 7ff6af363640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a8154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.899+0000 7ff6af363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a815afb0 0x7ff6a8155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.899+0000 7ff6af363640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a8157620 con 0x7ff6a80cb780 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.899+0000 7ff6af363640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a80cdd10 con 0x7ff6a815afb0 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.899+0000 7ff6af363640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a80cef10 con 0x7ff6a8154590 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.900+0000 7ff6ac8fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a8154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.900+0000 7ff6ad900640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a815afb0 0x7ff6a8155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.900+0000 7ff6ac8fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a8154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48836/0 (socket says 172.21.15.129:48836) 2024-03-23T15:47:41.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.900+0000 7ff6ac8fe640 1 -- 172.21.15.129:0/1475664537 learned_addr learned my addr 172.21.15.129:0/1475664537 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:41.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.904+0000 7ff6ac8fe640 1 -- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a815afb0 msgr2=0x7ff6a8155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.904+0000 7ff6ac8fe640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a815afb0 0x7ff6a8155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.904+0000 7ff6ac8fe640 1 -- 172.21.15.129:0/1475664537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a80cb780 msgr2=0x7ff6a814e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:41.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.904+0000 7ff6ac8fe640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a80cb780 0x7ff6a814e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:41.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.904+0000 7ff6ac8fe640 1 -- 172.21.15.129:0/1475664537 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a40094d0 con 0x7ff6a8154590 2024-03-23T15:47:41.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.906+0000 7ff6ac8fe640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a8154c40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff69800b4a0 tx=0x7ff69800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.911+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698010610 con 0x7ff6a8154590 2024-03-23T15:47:41.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.911+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff698010770 con 0x7ff6a8154590 2024-03-23T15:47:41.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.911+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a80cf1f0 con 0x7ff6a8154590 2024-03-23T15:47:41.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.911+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698010b20 con 0x7ff6a8154590 2024-03-23T15:47:41.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.911+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a8156fd0 con 0x7ff6a8154590 2024-03-23T15:47:41.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.913+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a80757b0 con 0x7ff6a8154590 2024-03-23T15:47:41.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.936+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff698010c80 con 0x7ff6a8154590 2024-03-23T15:47:41.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.936+0000 7ff69dffb640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff68c06b860 0x7ff68c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:41.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.936+0000 7ff6ad0ff640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff68c06b860 0x7ff68c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:41.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.937+0000 7ff6ad0ff640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff68c06b860 0x7ff68c06dd20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4036000 tx=0x7ff6a4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:41.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:41.939+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff6980589b0 con 0x7ff6a8154590 2024-03-23T15:47:41.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T15:47:42.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.077+0000 7f2413c15640 1 Processor -- start 2024-03-23T15:47:42.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.077+0000 7f2413c15640 1 -- start start 2024-03-23T15:47:42.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.078+0000 7f2413c15640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f240c0c93e0 0x7f240c15c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.078+0000 7f2413c15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240c15c780 con 0x7f240c158710 2024-03-23T15:47:42.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.078+0000 7f2413c15640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240c0c9d80 con 0x7f240c1525f0 2024-03-23T15:47:42.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.078+0000 7f2413c15640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f240c0c9030 con 0x7f240c0c93e0 2024-03-23T15:47:42.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.079+0000 7f24111b0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f240c0c93e0 0x7f240c15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.079+0000 7f24119b1640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f240c1525f0 0x7f240c0ca810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:39760/0 (socket says 172.21.15.129:39760) 2024-03-23T15:47:42.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.079+0000 7f24119b1640 1 -- 172.21.15.129:0/3202255153 learned_addr learned my addr 172.21.15.129:0/3202255153 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:42.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T15:47:42.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.083+0000 7f24109af640 1 -- 172.21.15.129:0/3202255153 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2858240514 0 0) 0x7f240c0c9d80 con 0x7f240c1525f0 2024-03-23T15:47:42.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.083+0000 7f24109af640 1 -- 172.21.15.129:0/3202255153 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f8003520 con 0x7f240c1525f0 2024-03-23T15:47:42.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.086+0000 7f24111b0640 1 -- 172.21.15.129:0/3202255153 >> v1:172.21.15.157:6789/0 conn(0x7f240c1525f0 legacy=0x7f240c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.086+0000 7f24111b0640 1 -- 172.21.15.129:0/3202255153 >> v1:172.21.15.129:6789/0 conn(0x7f240c158710 legacy=0x7f240c158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.086+0000 7f24111b0640 1 -- 172.21.15.129:0/3202255153 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c0ca060 con 0x7f240c0c93e0 2024-03-23T15:47:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.089+0000 7f24111b0640 1 --2- 172.21.15.129:0/3202255153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c0c93e0 0x7f240c15c240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0097a0 tx=0x7f23fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51bc05e5575f7f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.094+0000 7f24109af640 1 -- 172.21.15.129:0/3202255153 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc02e070 con 0x7f240c0c93e0 2024-03-23T15:47:42.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.094+0000 7f24109af640 1 -- 172.21.15.129:0/3202255153 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc0315a0 con 0x7f240c0c93e0 2024-03-23T15:47:42.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.094+0000 7f24109af640 1 -- 172.21.15.129:0/3202255153 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc031930 con 0x7f240c0c93e0 2024-03-23T15:47:42.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 -- 172.21.15.129:0/3202255153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c0c93e0 msgr2=0x7f240c15c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 --2- 172.21.15.129:0/3202255153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c0c93e0 0x7f240c15c240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0097a0 tx=0x7f23fc02b840 comp rx=0 tx=0).stop 2024-03-23T15:47:42.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 -- 172.21.15.129:0/3202255153 shutdown_connections 2024-03-23T15:47:42.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 --2- 172.21.15.129:0/3202255153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c0c93e0 0x7f240c15c240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 -- 172.21.15.129:0/3202255153 >> 172.21.15.129:0/3202255153 conn(0x7f240c071470 msgr2=0x7f240c071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 -- 172.21.15.129:0/3202255153 shutdown_connections 2024-03-23T15:47:42.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.095+0000 7f2413c15640 1 -- 172.21.15.129:0/3202255153 wait complete. 2024-03-23T15:47:42.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.097+0000 7f2413c15640 1 Processor -- start 2024-03-23T15:47:42.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.097+0000 7f2413c15640 1 -- start start 2024-03-23T15:47:42.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.098+0000 7f2413c15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f240c0c93e0 0x7f240c1eff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.098+0000 7f2413c15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 0x7f240c06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.098+0000 7f2413c15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f240c158710 0x7f240c0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.098+0000 7f2413c15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f240c0ae890 con 0x7f240c0c93e0 2024-03-23T15:47:42.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.098+0000 7f2413c15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f240c0af140 con 0x7f240c1525f0 2024-03-23T15:47:42.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.098+0000 7f2413c15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f240c06e2a0 con 0x7f240c158710 2024-03-23T15:47:42.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.099+0000 7f24111b0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 0x7f240c06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.099+0000 7f24111b0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 0x7f240c06dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52110/0 (socket says 172.21.15.129:52110) 2024-03-23T15:47:42.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.099+0000 7f24111b0640 1 -- 172.21.15.129:0/479531789 learned_addr learned my addr 172.21.15.129:0/479531789 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.104+0000 7f24111b0640 1 -- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f240c158710 msgr2=0x7f240c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.104+0000 7f24111b0640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f240c158710 0x7f240c0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.104+0000 7f24111b0640 1 -- 172.21.15.129:0/479531789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f240c0c93e0 msgr2=0x7f240c1eff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.104+0000 7f24111b0640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f240c0c93e0 0x7f240c1eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.104+0000 7f24111b0640 1 -- 172.21.15.129:0/479531789 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23fc009400 con 0x7f240c1525f0 2024-03-23T15:47:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.108+0000 7f24111b0640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 0x7f240c06dd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0098c0 tx=0x7f23fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.113+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0328c0 con 0x7f240c1525f0 2024-03-23T15:47:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.113+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc032a40 con 0x7f240c1525f0 2024-03-23T15:47:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.113+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240c06d8d0 con 0x7f240c1525f0 2024-03-23T15:47:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.113+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc032df0 con 0x7f240c1525f0 2024-03-23T15:47:42.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.113+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240c06d5e0 con 0x7f240c1525f0 2024-03-23T15:47:42.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.114+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f240c0757b0 con 0x7f240c1525f0 2024-03-23T15:47:42.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.146+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3504054440 con 0x7f3510070290 2024-03-23T15:47:42.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.188+0000 7f08c0ec0640 1 Processor -- start 2024-03-23T15:47:42.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.188+0000 7f08c0ec0640 1 -- start start 2024-03-23T15:47:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.189+0000 7f08c0ec0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f08bc0c3900 0x7f08bc0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.189+0000 7f08c0ec0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08bc0c8ca0 0x7f08bc0c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.189+0000 7f08c0ec0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc0ca220 con 0x7f08bc154580 2024-03-23T15:47:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.189+0000 7f08c0ec0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f08bc0c46b0 con 0x7f08bc0c3900 2024-03-23T15:47:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.189+0000 7f08c0ec0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f08bc15a420 con 0x7f08bc0c8ca0 2024-03-23T15:47:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.190+0000 7f08b3fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f08bc0c3900 0x7f08bc0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.190+0000 7f08b3fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f08bc0c3900 0x7f08bc0c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60678/0 (socket says 172.21.15.129:60678) 2024-03-23T15:47:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.190+0000 7f08b3fff640 1 -- 172.21.15.129:0/3936648302 learned_addr learned my addr 172.21.15.129:0/3936648302 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.193+0000 7f08b3fff640 1 -- 172.21.15.129:0/3936648302 >> v2:172.21.15.157:3300/0 conn(0x7f08bc0c8ca0 msgr2=0x7f08bc0c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.193+0000 7f08b3fff640 1 --2- 172.21.15.129:0/3936648302 >> v2:172.21.15.157:3300/0 conn(0x7f08bc0c8ca0 0x7f08bc0c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.193+0000 7f08b3fff640 1 -- 172.21.15.129:0/3936648302 >> v1:172.21.15.157:6789/0 conn(0x7f08bc154580 legacy=0x7f08bc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.193+0000 7f08b3fff640 1 -- 172.21.15.129:0/3936648302 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc154960 con 0x7f08bc0c3900 2024-03-23T15:47:42.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.195+0000 7f08b3fff640 1 --2- 172.21.15.129:0/3936648302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc0c3900 0x7f08bc0c8760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0c3820 tx=0x7f08a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a553b6d9a249d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.197+0000 7f08b9d9b640 1 -- 172.21.15.129:0/3936648302 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4031870 con 0x7f08bc0c3900 2024-03-23T15:47:42.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.199+0000 7f08b9d9b640 1 -- 172.21.15.129:0/3936648302 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a40319d0 con 0x7f08bc0c3900 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 -- 172.21.15.129:0/3936648302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc0c3900 msgr2=0x7f08bc0c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/3936648302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc0c3900 0x7f08bc0c8760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0c3820 tx=0x7f08a402b910 comp rx=0 tx=0).stop 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 -- 172.21.15.129:0/3936648302 shutdown_connections 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/3936648302 >> v2:172.21.15.157:3300/0 conn(0x7f08bc0c8ca0 0x7f08bc0c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/3936648302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc0c3900 0x7f08bc0c8760 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 -- 172.21.15.129:0/3936648302 >> 172.21.15.129:0/3936648302 conn(0x7f08bc071470 msgr2=0x7f08bc071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 -- 172.21.15.129:0/3936648302 shutdown_connections 2024-03-23T15:47:42.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.200+0000 7f08c0ec0640 1 -- 172.21.15.129:0/3936648302 wait complete. 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.202+0000 7f08c0ec0640 1 Processor -- start 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.202+0000 7f08c0ec0640 1 -- start start 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08c0ec0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08bc0c3900 0x7f08bc14ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08c0ec0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08bc0c8ca0 0x7f08bc14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08c0ec0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 0x7f08bc1f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08c0ec0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1f73e0 con 0x7f08bc154580 2024-03-23T15:47:42.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08c0ec0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1504c0 con 0x7f08bc0c3900 2024-03-23T15:47:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08c0ec0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08bc14dbe0 con 0x7f08bc0c8ca0 2024-03-23T15:47:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.203+0000 7f08bad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 0x7f08bc1f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.204+0000 7f08bad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 0x7f08bc1f3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60694/0 (socket says 172.21.15.129:60694) 2024-03-23T15:47:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.204+0000 7f08bad9d640 1 -- 172.21.15.129:0/2700909378 learned_addr learned my addr 172.21.15.129:0/2700909378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:42.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.209+0000 7f08bad9d640 1 -- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08bc0c8ca0 msgr2=0x7f08bc14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.209+0000 7f08bad9d640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08bc0c8ca0 0x7f08bc14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.209+0000 7f08bad9d640 1 -- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08bc0c3900 msgr2=0x7f08bc14ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.209+0000 7f08bad9d640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08bc0c3900 0x7f08bc14ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.209+0000 7f08bad9d640 1 -- 172.21.15.129:0/2700909378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a40094d0 con 0x7f08bc154580 2024-03-23T15:47:42.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.213+0000 7f08bad9d640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 0x7f08bc1f3e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08b400b970 tx=0x7f08b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.218+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b400cb40 con 0x7f08bc154580 2024-03-23T15:47:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.218+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b400ccc0 con 0x7f08bc154580 2024-03-23T15:47:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.218+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4014680 con 0x7f08bc154580 2024-03-23T15:47:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.218+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08bc14dec0 con 0x7f08bc154580 2024-03-23T15:47:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.218+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08bc1f7910 con 0x7f08bc154580 2024-03-23T15:47:42.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.219+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08bc0757b0 con 0x7f08bc154580 2024-03-23T15:47:42.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T15:47:42.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.358+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f351014cc40 con 0x7f3510070290 2024-03-23T15:47:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.380+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff698054c50 con 0x7ff6a8154590 2024-03-23T15:47:42.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.402+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f23fc0453f0 con 0x7f240c1525f0 2024-03-23T15:47:42.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.402+0000 7f24027fc640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f23f006b810 0x7f23f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.403+0000 7f24119b1640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f23f006b810 0x7f23f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.403+0000 7f24119b1640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f23f006b810 0x7f23f006dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2408000f60 tx=0x7f24080078f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.407+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f23fc07b180 con 0x7f240c1525f0 2024-03-23T15:47:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.420+0000 7f4f2e283640 1 Processor -- start 2024-03-23T15:47:42.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.420+0000 7f4f2e283640 1 -- start start 2024-03-23T15:47:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.421+0000 7f4f2e283640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f280c7700 0x7f4f2815d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.421+0000 7f4f2e283640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f280cc2a0 con 0x7f4f28154590 2024-03-23T15:47:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.421+0000 7f4f2e283640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2815ab20 con 0x7f4f280cb780 2024-03-23T15:47:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.421+0000 7f4f2e283640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f4f280cbce0 con 0x7f4f280c7700 2024-03-23T15:47:42.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.442+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f23fc077420 con 0x7f240c1525f0 2024-03-23T15:47:42.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.452+0000 7f4f277fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4f280cb780 0x7f4f280c6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:39768/0 (socket says 172.21.15.129:39768) 2024-03-23T15:47:42.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.452+0000 7f4f277fe640 1 -- 172.21.15.129:0/1992552596 learned_addr learned my addr 172.21.15.129:0/1992552596 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:42.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.470+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478560901 0 0) 0x7f4f2815ab20 con 0x7f4f280cb780 2024-03-23T15:47:42.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.470+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f10003520 con 0x7f4f280cb780 2024-03-23T15:47:42.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3445190890 0 0) 0x7f4f10003520 con 0x7f4f280cb780 2024-03-23T15:47:42.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 >> v2:172.21.15.129:3300/0 conn(0x7f4f280c7700 msgr2=0x7f4f2815d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 --2- 172.21.15.129:0/1992552596 >> v2:172.21.15.129:3300/0 conn(0x7f4f280c7700 0x7f4f2815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 >> v1:172.21.15.129:6789/0 conn(0x7f4f28154590 legacy=0x7f4f280ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f280cbfc0 con 0x7f4f280cb780 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f4f1c001670 con 0x7f4f280cb780 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 >> v1:172.21.15.157:6789/0 conn(0x7f4f280cb780 legacy=0x7f4f280c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f10003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f100047c0 0x7f4f10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f10005380 0x7f4f1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280cbce0 con 0x7f4f100047c0 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1000c220 con 0x7f4f10005380 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.475+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1000c3e0 con 0x7f4f10003970 2024-03-23T15:47:42.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.476+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f10003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.477+0000 7f4f2c820640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f10005380 0x7f4f1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.480+0000 7f4f27fff640 1 -- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f10005380 msgr2=0x7f4f1000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.480+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f10005380 0x7f4f1000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.480+0000 7f4f27fff640 1 -- 172.21.15.129:0/1992552596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f100047c0 msgr2=0x7f4f10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.480+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f100047c0 0x7f4f10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.480+0000 7f4f27fff640 1 -- 172.21.15.129:0/1992552596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f280cbfc0 con 0x7f4f10003970 2024-03-23T15:47:42.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.480+0000 7f4f2c820640 1 -- 172.21.15.129:0/1992552596 reap_dead start 2024-03-23T15:47:42.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.483+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f10003d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f18001650 tx=0x7f4f100057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad87090442d481e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.487+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18005270 con 0x7f4f10003970 2024-03-23T15:47:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.487+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f1802e070 con 0x7f4f10003970 2024-03-23T15:47:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.487+0000 7f4f26ffd640 1 -- 172.21.15.129:0/1992552596 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18003610 con 0x7f4f10003970 2024-03-23T15:47:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.488+0000 7f4f2e283640 1 -- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 msgr2=0x7f4f10003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.488+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f10003d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f18001650 tx=0x7f4f100057f0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.488+0000 7f4f2e283640 1 -- 172.21.15.129:0/1992552596 shutdown_connections 2024-03-23T15:47:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.489+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1992552596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f10003d70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.489+0000 7f4f2e283640 1 -- 172.21.15.129:0/1992552596 >> 172.21.15.129:0/1992552596 conn(0x7f4f28061b50 msgr2=0x7f4f28061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.489+0000 7f4f2e283640 1 -- 172.21.15.129:0/1992552596 shutdown_connections 2024-03-23T15:47:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.489+0000 7f4f2e283640 1 -- 172.21.15.129:0/1992552596 wait complete. 2024-03-23T15:47:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.491+0000 7f4f2e283640 1 Processor -- start 2024-03-23T15:47:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.491+0000 7f4f2e283640 1 -- start start 2024-03-23T15:47:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.492+0000 7f4f2e283640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f2814d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.492+0000 7f4f2e283640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f281e5720 0x7f4f281e2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.492+0000 7f4f2e283640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f2815c750 0x7f4f2815cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.492+0000 7f4f2e283640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2815d110 con 0x7f4f281e5720 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.492+0000 7f4f2e283640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f281e45f0 con 0x7f4f2815c750 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.492+0000 7f4f2e283640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f281e3620 con 0x7f4f10003970 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.493+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f2814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.493+0000 7f4f2c820640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f2815c750 0x7f4f2815cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.493+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f2814d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48874/0 (socket says 172.21.15.129:48874) 2024-03-23T15:47:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.493+0000 7f4f27fff640 1 -- 172.21.15.129:0/1578565098 learned_addr learned my addr 172.21.15.129:0/1578565098 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.497+0000 7f4f27fff640 1 -- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f2815c750 msgr2=0x7f4f2815cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.497+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f2815c750 0x7f4f2815cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.497+0000 7f4f27fff640 1 -- 172.21.15.129:0/1578565098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f281e5720 msgr2=0x7f4f281e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.497+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f281e5720 0x7f4f281e2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.497+0000 7f4f27fff640 1 -- 172.21.15.129:0/1578565098 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f281efb60 con 0x7f4f10003970 2024-03-23T15:47:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.500+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f08b401d3b0 con 0x7f08bc154580 2024-03-23T15:47:42.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.500+0000 7f4f27fff640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f2814d210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f18001770 tx=0x7f4f18000fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.500+0000 7f08b2ffd640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f089c06b890 0x7f089c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.500+0000 7f08ba59c640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f089c06b890 0x7f089c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.501+0000 7f08ba59c640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f089c06b890 0x7f089c06dd50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009650 tx=0x7f08ac006ba0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.503+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18007240 con 0x7f4f10003970 2024-03-23T15:47:42.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.503+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f281efe40 con 0x7f4f10003970 2024-03-23T15:47:42.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.504+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f281f0400 con 0x7f4f10003970 2024-03-23T15:47:42.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.504+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f180073a0 con 0x7f4f10003970 2024-03-23T15:47:42.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.504+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f180067b0 con 0x7f4f10003970 2024-03-23T15:47:42.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.505+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f08005290 con 0x7f4f10003970 2024-03-23T15:47:42.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.515+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f08b400ce30 con 0x7f08bc154580 2024-03-23T15:47:42.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.515+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08b4055010 con 0x7f08bc154580 2024-03-23T15:47:42.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.530+0000 7f34feffd640 1 -- 172.21.15.129:0/3355493863 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f351014cc40 con 0x7f3510070290 2024-03-23T15:47:42.533 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f34f006b890 msgr2=0x7f34f006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f34f006b890 0x7f34f006dd50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3500009aa0 tx=0x7f3500009190 comp rx=0 tx=0).stop 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 msgr2=0x7f351014cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351014cf20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3504000f60 tx=0x7f350400c800 comp rx=0 tx=0).stop 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 shutdown_connections 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3510158750 0x7f351014d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3510070290 0x7f351014cf20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f34f006b890 0x7f34f006dd50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 --2- 172.21.15.129:0/3355493863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f351006fe50 0x7f35101f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 >> 172.21.15.129:0/3355493863 conn(0x7f35100622c0 msgr2=0x7f35100c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 shutdown_connections 2024-03-23T15:47:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.533+0000 7f350f59e640 1 -- 172.21.15.129:0/3355493863 wait complete. 2024-03-23T15:47:42.553 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-03-23T15:47:42.553 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:42.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T15:47:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.628+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff6a8157790 con 0x7ff6a8154590 2024-03-23T15:47:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.686+0000 7faaa8484640 1 Processor -- start 2024-03-23T15:47:42.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.686+0000 7faaa8484640 1 -- start start 2024-03-23T15:47:42.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.686+0000 7faaa8484640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 0x7faaa015d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.687+0000 7faaa8484640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faaa015afb0 0x7faaa00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.687+0000 7faaa8484640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa00cc2a0 con 0x7faaa00cb780 2024-03-23T15:47:42.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.687+0000 7faaa8484640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faaa015ab20 con 0x7faaa015afb0 2024-03-23T15:47:42.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.687+0000 7faaa8484640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faaa00cbce0 con 0x7faaa0154590 2024-03-23T15:47:42.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.687+0000 7faaa6a21640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faaa015afb0 0x7faaa00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.688+0000 7faaa6a21640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faaa015afb0 0x7faaa00c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52174/0 (socket says 172.21.15.129:52174) 2024-03-23T15:47:42.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.688+0000 7faaa6a21640 1 -- 172.21.15.129:0/248048404 learned_addr learned my addr 172.21.15.129:0/248048404 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:42.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.692+0000 7faaa6a21640 1 -- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 msgr2=0x7faaa015d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:42.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.692+0000 7faaa6a21640 1 --2- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 0x7faaa015d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.692+0000 7faaa6a21640 1 -- 172.21.15.129:0/248048404 >> v1:172.21.15.157:6789/0 conn(0x7faaa00cb780 legacy=0x7faaa00ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.692+0000 7faaa6a21640 1 -- 172.21.15.129:0/248048404 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa00cbfc0 con 0x7faaa015afb0 2024-03-23T15:47:42.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.698+0000 7faaa6a21640 1 --2- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa015afb0 0x7faaa00c6ff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa98009e50 tx=0x7faa9802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58587cd4d17675ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.704+0000 7faaa521e640 1 -- 172.21.15.129:0/248048404 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9802ca70 con 0x7faaa015afb0 2024-03-23T15:47:42.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.704+0000 7faaa521e640 1 -- 172.21.15.129:0/248048404 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9802cbd0 con 0x7faaa015afb0 2024-03-23T15:47:42.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.704+0000 7faaa521e640 1 -- 172.21.15.129:0/248048404 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa980343f0 con 0x7faaa015afb0 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 -- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa015afb0 msgr2=0x7faaa00c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 --2- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa015afb0 0x7faaa00c6ff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa98009e50 tx=0x7faa9802bed0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 -- 172.21.15.129:0/248048404 shutdown_connections 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 --2- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa015afb0 0x7faaa00c6ff0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 --2- 172.21.15.129:0/248048404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 0x7faaa015d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 -- 172.21.15.129:0/248048404 >> 172.21.15.129:0/248048404 conn(0x7faaa0061b50 msgr2=0x7faaa0061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 -- 172.21.15.129:0/248048404 shutdown_connections 2024-03-23T15:47:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.705+0000 7faaa8484640 1 -- 172.21.15.129:0/248048404 wait complete. 2024-03-23T15:47:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.706+0000 7faaa8484640 1 Processor -- start 2024-03-23T15:47:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.706+0000 7faaa8484640 1 -- start start 2024-03-23T15:47:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.707+0000 7faaa8484640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa00cb780 0x7faaa01ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.707+0000 7faaa8484640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 0x7faaa015cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.707+0000 7faaa8484640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 0x7faaa0156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.707+0000 7faaa8484640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faaa0154d70 con 0x7faaa015afb0 2024-03-23T15:47:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.707+0000 7faaa8484640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faaa0155050 con 0x7faaa00cb780 2024-03-23T15:47:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.707+0000 7faaa8484640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faaa0155330 con 0x7faaa0154590 2024-03-23T15:47:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.708+0000 7faaa6a21640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 0x7faaa0156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.708+0000 7faaa6220640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa00cb780 0x7faaa01ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.708+0000 7faaa6a21640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 0x7faaa0156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60732/0 (socket says 172.21.15.129:60732) 2024-03-23T15:47:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.708+0000 7faaa6a21640 1 -- 172.21.15.129:0/37803399 learned_addr learned my addr 172.21.15.129:0/37803399 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.713+0000 7faaa6a21640 1 -- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 msgr2=0x7faaa015cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.713+0000 7faaa6a21640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 0x7faaa015cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.713+0000 7faaa6a21640 1 -- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa00cb780 msgr2=0x7faaa01ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.713+0000 7faaa6a21640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa00cb780 0x7faaa01ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.713+0000 7faaa6a21640 1 -- 172.21.15.129:0/37803399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa98009b30 con 0x7faaa015afb0 2024-03-23T15:47:42.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.716+0000 7faaa6a21640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 0x7faaa0156380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faa98036040 tx=0x7faa98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.720+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98040670 con 0x7faaa015afb0 2024-03-23T15:47:42.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.720+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa980407f0 con 0x7faaa015afb0 2024-03-23T15:47:42.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.720+0000 7faaa8484640 1 -- 172.21.15.129:0/37803399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa0155610 con 0x7faaa015afb0 2024-03-23T15:47:42.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.720+0000 7faaa8484640 1 -- 172.21.15.129:0/37803399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa0155bd0 con 0x7faaa015afb0 2024-03-23T15:47:42.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.720+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98040ba0 con 0x7faaa015afb0 2024-03-23T15:47:42.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.722+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaa01560a0 con 0x7faaa015afb0 2024-03-23T15:47:42.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.741+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f08bc0c91a0 con 0x7f08bc154580 2024-03-23T15:47:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.752+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faa98040d00 con 0x7faaa015afb0 2024-03-23T15:47:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.752+0000 7faa96ffd640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faa8406b860 0x7faa8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.753+0000 7faaa6220640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faa8406b860 0x7faa8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.753+0000 7faaa6220640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faa8406b860 0x7faa8406dd20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009990 tx=0x7faa9c009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.772+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f18030030 con 0x7f4f10003970 2024-03-23T15:47:42.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.773+0000 7f4f24ff9640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f1406b840 0x7f4f1406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:42.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.773+0000 7f4f277fe640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f1406b840 0x7f4f1406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:42.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.774+0000 7f4f277fe640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f1406b840 0x7f4f1406dd00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f2814fe60 tx=0x7f4f1c001600 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:42.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.777+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4f1807a120 con 0x7f4f10003970 2024-03-23T15:47:42.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.792+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f240c15c780 con 0x7f240c1525f0 2024-03-23T15:47:42.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.796+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f18075cc0 con 0x7f4f10003970 2024-03-23T15:47:42.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.798+0000 7f24027fc640 1 -- 172.21.15.129:0/479531789 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f240c15c780 con 0x7f240c1525f0 2024-03-23T15:47:42.801 INFO:teuthology.orchestra.run.smithi129.stdout:55834574856 2024-03-23T15:47:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f23f006b810 msgr2=0x7f23f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f23f006b810 0x7f23f006dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2408000f60 tx=0x7f24080078f0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 msgr2=0x7f240c06dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 0x7f240c06dd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0098c0 tx=0x7f23fc0046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 shutdown_connections 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f240c158710 0x7f240c0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f23f006b810 0x7f23f006dcd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f240c1525f0 0x7f240c06dd60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 --2- 172.21.15.129:0/479531789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f240c0c93e0 0x7f240c1eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.800+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 >> 172.21.15.129:0/479531789 conn(0x7f240c071470 msgr2=0x7f240c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.801+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 shutdown_connections 2024-03-23T15:47:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.801+0000 7f2413c15640 1 -- 172.21.15.129:0/479531789 wait complete. 2024-03-23T15:47:42.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.812+0000 7ff69dffb640 1 -- 172.21.15.129:0/1475664537 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff69801e070 con 0x7ff6a8154590 2024-03-23T15:47:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:42.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-03-23T15:47:42.816 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff68c06b860 msgr2=0x7ff68c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff68c06b860 0x7ff68c06dd20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4036000 tx=0x7ff6a4030000 comp rx=0 tx=0).stop 2024-03-23T15:47:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 msgr2=0x7ff6a8154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a8154c40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff69800b4a0 tx=0x7ff69800b960 comp rx=0 tx=0).stop 2024-03-23T15:47:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 shutdown_connections 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff68c06b860 0x7ff68c06dd20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a815afb0 0x7ff6a8155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6a8154590 0x7ff6a8154c40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 --2- 172.21.15.129:0/1475664537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a80cb780 0x7ff6a814e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.814+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 >> 172.21.15.129:0/1475664537 conn(0x7ff6a8061b50 msgr2=0x7ff6a814fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.815+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 shutdown_connections 2024-03-23T15:47:42.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:42.815+0000 7ff6af363640 1 -- 172.21.15.129:0/1475664537 wait complete. 2024-03-23T15:47:42.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-03-23T15:47:42.829 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:43.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.007+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faa9807c510 con 0x7faaa015afb0 2024-03-23T15:47:43.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.007+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faa980780b0 con 0x7faaa015afb0 2024-03-23T15:47:43.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.016+0000 7f08b2ffd640 1 -- 172.21.15.129:0/2700909378 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f08b4019030 con 0x7f08bc154580 2024-03-23T15:47:43.019 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:43.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.019+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f089c06b890 msgr2=0x7f089c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f089c06b890 0x7f089c06dd50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009650 tx=0x7f08ac006ba0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 msgr2=0x7f08bc1f3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 0x7f08bc1f3e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08b400b970 tx=0x7f08b400be30 comp rx=0 tx=0).stop 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 shutdown_connections 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08bc154580 0x7f08bc1f3e70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08bc0c8ca0 0x7f08bc14d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f089c06b890 0x7f089c06dd50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 --2- 172.21.15.129:0/2700909378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08bc0c3900 0x7f08bc14ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 >> 172.21.15.129:0/2700909378 conn(0x7f08bc071470 msgr2=0x7f08bc1538a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 shutdown_connections 2024-03-23T15:47:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.020+0000 7f08c0ec0640 1 -- 172.21.15.129:0/2700909378 wait complete. 2024-03-23T15:47:43.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.026+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4f08005810 con 0x7f4f10003970 2024-03-23T15:47:43.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-03-23T15:47:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.036+0000 7f4f24ff9640 1 -- 172.21.15.129:0/1578565098 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4f18079380 con 0x7f4f10003970 2024-03-23T15:47:43.039 INFO:teuthology.orchestra.run.smithi129.stdout:55834574857 2024-03-23T15:47:43.039 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f1406b840 msgr2=0x7f4f1406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f1406b840 0x7f4f1406dd00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f2814fe60 tx=0x7f4f1c001600 comp rx=0 tx=0).stop 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 msgr2=0x7f4f2814d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f2814d210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f18001770 tx=0x7f4f18000fb0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 shutdown_connections 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f1406b840 0x7f4f1406dd00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f2815c750 0x7f4f2815cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f281e5720 0x7f4f281e2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 --2- 172.21.15.129:0/1578565098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f10003970 0x7f4f2814d210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 >> 172.21.15.129:0/1578565098 conn(0x7f4f28061b50 msgr2=0x7f4f28150470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 shutdown_connections 2024-03-23T15:47:43.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.039+0000 7f4f2e283640 1 -- 172.21.15.129:0/1578565098 wait complete. 2024-03-23T15:47:43.053 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-03-23T15:47:43.053 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:43.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.237+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7faaa01560a0 con 0x7faaa015afb0 2024-03-23T15:47:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.243+0000 7faa96ffd640 1 -- 172.21.15.129:0/37803399 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faaa01560a0 con 0x7faaa015afb0 2024-03-23T15:47:43.246 INFO:teuthology.orchestra.run.smithi129.stdout:55834574858 2024-03-23T15:47:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faa8406b860 msgr2=0x7faa8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faa8406b860 0x7faa8406dd20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009990 tx=0x7faa9c009110 comp rx=0 tx=0).stop 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 msgr2=0x7faaa0156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 0x7faaa0156380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faa98036040 tx=0x7faa98004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 shutdown_connections 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa015afb0 0x7faaa0156380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faaa0154590 0x7faaa015cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faa8406b860 0x7faa8406dd20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 --2- 172.21.15.129:0/37803399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faaa00cb780 0x7faaa01ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 >> 172.21.15.129:0/37803399 conn(0x7faaa0061b50 msgr2=0x7faaa0150330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 shutdown_connections 2024-03-23T15:47:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.245+0000 7faa94ff9640 1 -- 172.21.15.129:0/37803399 wait complete. 2024-03-23T15:47:43.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.4 2024-03-23T15:47:43.259 DEBUG:teuthology.parallel:result is None 2024-03-23T15:47:43.259 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T15:47:43.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T15:47:43.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.359+0000 7f20d50c2640 1 Processor -- start 2024-03-23T15:47:43.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.359+0000 7f20d50c2640 1 -- start start 2024-03-23T15:47:43.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.360+0000 7f20d50c2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f20d015a2e0 0x7f20d006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:43.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.360+0000 7f20d50c2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d00cbcb0 con 0x7f20d01544b0 2024-03-23T15:47:43.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.360+0000 7f20d50c2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d015b280 con 0x7f20d006c0b0 2024-03-23T15:47:43.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.360+0000 7f20d50c2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f20d00c5be0 con 0x7f20d015a2e0 2024-03-23T15:47:43.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.361+0000 7f20ced9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f20d01544b0 0x7f20d00caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44698/0 (socket says 172.21.15.129:44698) 2024-03-23T15:47:43.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.361+0000 7f20ced9d640 1 -- 172.21.15.129:0/2006101411 learned_addr learned my addr 172.21.15.129:0/2006101411 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:43.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.361+0000 7f20ce59c640 1 --2- 172.21.15.129:0/2006101411 >> v2:172.21.15.157:3300/0 conn(0x7f20d015a2e0 0x7f20d006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:43.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.364+0000 7f20cdd9b640 1 -- 172.21.15.129:0/2006101411 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3355739327 0 0) 0x7f20d00cbcb0 con 0x7f20d01544b0 2024-03-23T15:47:43.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.364+0000 7f20cdd9b640 1 -- 172.21.15.129:0/2006101411 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20b8003520 con 0x7f20d01544b0 2024-03-23T15:47:43.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.365+0000 7f20ce59c640 1 -- 172.21.15.129:0/2006101411 >> v1:172.21.15.157:6789/0 conn(0x7f20d006c0b0 legacy=0x7f20d00c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.365+0000 7f20ce59c640 1 -- 172.21.15.129:0/2006101411 >> v1:172.21.15.129:6789/0 conn(0x7f20d01544b0 legacy=0x7f20d00caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.365+0000 7f20ce59c640 1 -- 172.21.15.129:0/2006101411 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d015a9e0 con 0x7f20d015a2e0 2024-03-23T15:47:43.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.374+0000 7f20ce59c640 1 --2- 172.21.15.129:0/2006101411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d006bb70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20c40097a0 tx=0x7f20c402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f09e1870f438ac9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:43.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.377+0000 7f20cdd9b640 1 -- 172.21.15.129:0/2006101411 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c402f810 con 0x7f20d015a2e0 2024-03-23T15:47:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.381+0000 7f20cdd9b640 1 -- 172.21.15.129:0/2006101411 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c402f970 con 0x7f20d015a2e0 2024-03-23T15:47:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.381+0000 7f20cdd9b640 1 -- 172.21.15.129:0/2006101411 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c402fce0 con 0x7f20d015a2e0 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 -- 172.21.15.129:0/2006101411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 msgr2=0x7f20d006bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 --2- 172.21.15.129:0/2006101411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d006bb70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20c40097a0 tx=0x7f20c402b9f0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 -- 172.21.15.129:0/2006101411 shutdown_connections 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 --2- 172.21.15.129:0/2006101411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d006bb70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 -- 172.21.15.129:0/2006101411 >> 172.21.15.129:0/2006101411 conn(0x7f20d0061b50 msgr2=0x7f20d0061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 -- 172.21.15.129:0/2006101411 shutdown_connections 2024-03-23T15:47:43.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.382+0000 7f20d50c2640 1 -- 172.21.15.129:0/2006101411 wait complete. 2024-03-23T15:47:43.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 Processor -- start 2024-03-23T15:47:43.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 -- start start 2024-03-23T15:47:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 0x7f20d01576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20d01544b0 0x7f20d01f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d00c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20d01420a0 con 0x7f20d01544b0 2024-03-23T15:47:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f20d01f0b30 con 0x7f20d015a2e0 2024-03-23T15:47:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.384+0000 7f20d50c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f20d01f4600 con 0x7f20d006c0b0 2024-03-23T15:47:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.385+0000 7f20ced9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 0x7f20d01576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.385+0000 7f20ce59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20d01544b0 0x7f20d01f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.385+0000 7f20ced9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 0x7f20d01576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48900/0 (socket says 172.21.15.129:48900) 2024-03-23T15:47:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.385+0000 7f20ced9d640 1 -- 172.21.15.129:0/3753264146 learned_addr learned my addr 172.21.15.129:0/3753264146 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.387+0000 7f20cf59e640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d00c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:43.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.389+0000 7f20ced9d640 1 -- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 msgr2=0x7f20d00c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.389+0000 7f20ced9d640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d00c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.389+0000 7f20ced9d640 1 -- 172.21.15.129:0/3753264146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20d01544b0 msgr2=0x7f20d01f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.389+0000 7f20ced9d640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20d01544b0 0x7f20d01f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.389+0000 7f20ced9d640 1 -- 172.21.15.129:0/3753264146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c4009400 con 0x7f20d006c0b0 2024-03-23T15:47:43.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.391+0000 7f20ced9d640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 0x7f20d01576a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c500 tx=0x7f20c000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:43.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.396+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0005ce0 con 0x7f20d006c0b0 2024-03-23T15:47:43.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.396+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c0005e40 con 0x7f20d006c0b0 2024-03-23T15:47:43.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.396+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0007330 con 0x7f20d006c0b0 2024-03-23T15:47:43.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.396+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d0147100 con 0x7f20d006c0b0 2024-03-23T15:47:43.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.396+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d01f04e0 con 0x7f20d006c0b0 2024-03-23T15:47:43.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.397+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d0075740 con 0x7f20d006c0b0 2024-03-23T15:47:43.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.421+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f20c0007550 con 0x7f20d006c0b0 2024-03-23T15:47:43.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.422+0000 7f20b77fe640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f20bc06b860 0x7f20bc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:43.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.422+0000 7f20ce59c640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f20bc06b860 0x7f20bc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:43.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.423+0000 7f20ce59c640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f20bc06b860 0x7f20bc06dd20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20c4036000 tx=0x7f20c403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.424+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f20c0058570 con 0x7f20d006c0b0 2024-03-23T15:47:43.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.692+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f20c0054810 con 0x7f20d006c0b0 2024-03-23T15:47:43.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.870+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f20d0158190 con 0x7f20bc06b860 2024-03-23T15:47:43.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.883+0000 7f20b77fe640 1 -- 172.21.15.129:0/3753264146 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252041 (secure 0 0 0) 0x7f20d0158190 con 0x7f20bc06b860 2024-03-23T15:47:43.888 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:43.889 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f20bc06b860 msgr2=0x7f20bc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f20bc06b860 0x7f20bc06dd20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20c4036000 tx=0x7f20c403b040 comp rx=0 tx=0).stop 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 msgr2=0x7f20d01576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 0x7f20d01576a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c500 tx=0x7f20c000c9c0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 shutdown_connections 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f20bc06b860 0x7f20bc06dd20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20d015a2e0 0x7f20d00c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20d01544b0 0x7f20d01f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 --2- 172.21.15.129:0/3753264146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20d006c0b0 0x7f20d01576a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:43.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 >> 172.21.15.129:0/3753264146 conn(0x7f20d0061b50 msgr2=0x7f20d0151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:43.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 shutdown_connections 2024-03-23T15:47:43.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:43.886+0000 7f20d50c2640 1 -- 172.21.15.129:0/3753264146 wait complete. 2024-03-23T15:47:43.900 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-23T15:47:42.408861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":362,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230688,"kb_used_data":2080,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754744032,"statfs":{"total":773094113280,"available":772857888768,"internally_reserved":0,"allocated":2129920,"data_stored":1804732,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8034,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002591"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686665+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:47:28.686665+0000","last_peered":"2024-03-23T15:47:28.686665+0000","last_clean":"2024-03-23T15:47:28.686665+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:47:28.686665+0000","last_undegraded":"2024-03-23T15:47:28.686665+0000","last_fullsized":"2024-03-23T15:47:28.686665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411203+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:47:28.411203+0000","last_peered":"2024-03-23T15:47:28.411203+0000","last_clean":"2024-03-23T15:47:28.411203+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:47:28.411203+0000","last_undegraded":"2024-03-23T15:47:28.411203+0000","last_fullsized":"2024-03-23T15:47:28.411203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411144+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:47:28.411144+0000","last_peered":"2024-03-23T15:47:28.411144+0000","last_clean":"2024-03-23T15:47:28.411144+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:47:28.411144+0000","last_undegraded":"2024-03-23T15:47:28.411144+0000","last_fullsized":"2024-03-23T15:47:28.411144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682515+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:47:28.682515+0000","last_peered":"2024-03-23T15:47:28.682515+0000","last_clean":"2024-03-23T15:47:28.682515+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:47:28.682515+0000","last_undegraded":"2024-03-23T15:47:28.682515+0000","last_fullsized":"2024-03-23T15:47:28.682515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686424+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:47:28.686424+0000","last_peered":"2024-03-23T15:47:28.686424+0000","last_clean":"2024-03-23T15:47:28.686424+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:47:28.686424+0000","last_undegraded":"2024-03-23T15:47:28.686424+0000","last_fullsized":"2024-03-23T15:47:28.686424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682361+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:47:28.682361+0000","last_peered":"2024-03-23T15:47:28.682361+0000","last_clean":"2024-03-23T15:47:28.682361+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:47:28.682361+0000","last_undegraded":"2024-03-23T15:47:28.682361+0000","last_fullsized":"2024-03-23T15:47:28.682361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686125+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:47:28.686125+0000","last_peered":"2024-03-23T15:47:28.686125+0000","last_clean":"2024-03-23T15:47:28.686125+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:47:28.686125+0000","last_undegraded":"2024-03-23T15:47:28.686125+0000","last_fullsized":"2024-03-23T15:47:28.686125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410933+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:47:28.410933+0000","last_peered":"2024-03-23T15:47:28.410933+0000","last_clean":"2024-03-23T15:47:28.410933+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:47:28.410933+0000","last_undegraded":"2024-03-23T15:47:28.410933+0000","last_fullsized":"2024-03-23T15:47:28.410933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465785+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:47:28.465785+0000","last_peered":"2024-03-23T15:47:28.465785+0000","last_clean":"2024-03-23T15:47:28.465785+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:47:28.465785+0000","last_undegraded":"2024-03-23T15:47:28.465785+0000","last_fullsized":"2024-03-23T15:47:28.465785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686368+0000","last_change":"2024-03-23T15:47:27.453034+0000","last_active":"2024-03-23T15:47:28.686368+0000","last_peered":"2024-03-23T15:47:28.686368+0000","last_clean":"2024-03-23T15:47:28.686368+0000","last_became_active":"2024-03-23T15:47:27.452786+0000","last_became_peered":"2024-03-23T15:47:27.452786+0000","last_unstale":"2024-03-23T15:47:28.686368+0000","last_undegraded":"2024-03-23T15:47:28.686368+0000","last_fullsized":"2024-03-23T15:47:28.686368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747756+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:47:28.747756+0000","last_peered":"2024-03-23T15:47:28.747756+0000","last_clean":"2024-03-23T15:47:28.747756+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:47:28.747756+0000","last_undegraded":"2024-03-23T15:47:28.747756+0000","last_fullsized":"2024-03-23T15:47:28.747756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747609+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:47:28.747609+0000","last_peered":"2024-03-23T15:47:28.747609+0000","last_clean":"2024-03-23T15:47:28.747609+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:47:28.747609+0000","last_undegraded":"2024-03-23T15:47:28.747609+0000","last_fullsized":"2024-03-23T15:47:28.747609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410854+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:47:28.410854+0000","last_peered":"2024-03-23T15:47:28.410854+0000","last_clean":"2024-03-23T15:47:28.410854+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:47:28.410854+0000","last_undegraded":"2024-03-23T15:47:28.410854+0000","last_fullsized":"2024-03-23T15:47:28.410854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416608+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:47:28.416608+0000","last_peered":"2024-03-23T15:47:28.416608+0000","last_clean":"2024-03-23T15:47:28.416608+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:47:28.416608+0000","last_undegraded":"2024-03-23T15:47:28.416608+0000","last_fullsized":"2024-03-23T15:47:28.416608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417524+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:47:28.417524+0000","last_peered":"2024-03-23T15:47:28.417524+0000","last_clean":"2024-03-23T15:47:28.417524+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:47:28.417524+0000","last_undegraded":"2024-03-23T15:47:28.417524+0000","last_fullsized":"2024-03-23T15:47:28.417524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681899+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:47:28.681899+0000","last_peered":"2024-03-23T15:47:28.681899+0000","last_clean":"2024-03-23T15:47:28.681899+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:47:28.681899+0000","last_undegraded":"2024-03-23T15:47:28.681899+0000","last_fullsized":"2024-03-23T15:47:28.681899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417419+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:47:28.417419+0000","last_peered":"2024-03-23T15:47:28.417419+0000","last_clean":"2024-03-23T15:47:28.417419+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:47:28.417419+0000","last_undegraded":"2024-03-23T15:47:28.417419+0000","last_fullsized":"2024-03-23T15:47:28.417419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686539+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:47:28.686539+0000","last_peered":"2024-03-23T15:47:28.686539+0000","last_clean":"2024-03-23T15:47:28.686539+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:47:28.686539+0000","last_undegraded":"2024-03-23T15:47:28.686539+0000","last_fullsized":"2024-03-23T15:47:28.686539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679986+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:47:28.679986+0000","last_peered":"2024-03-23T15:47:28.679986+0000","last_clean":"2024-03-23T15:47:28.679986+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:47:28.679986+0000","last_undegraded":"2024-03-23T15:47:28.679986+0000","last_fullsized":"2024-03-23T15:47:28.679986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465648+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:47:28.465648+0000","last_peered":"2024-03-23T15:47:28.465648+0000","last_clean":"2024-03-23T15:47:28.465648+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:47:28.465648+0000","last_undegraded":"2024-03-23T15:47:28.465648+0000","last_fullsized":"2024-03-23T15:47:28.465648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410906+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:47:28.410906+0000","last_peered":"2024-03-23T15:47:28.410906+0000","last_clean":"2024-03-23T15:47:28.410906+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:47:28.410906+0000","last_undegraded":"2024-03-23T15:47:28.410906+0000","last_fullsized":"2024-03-23T15:47:28.410906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416609+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:47:28.416609+0000","last_peered":"2024-03-23T15:47:28.416609+0000","last_clean":"2024-03-23T15:47:28.416609+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:47:28.416609+0000","last_undegraded":"2024-03-23T15:47:28.416609+0000","last_fullsized":"2024-03-23T15:47:28.416609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411310+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:47:28.411310+0000","last_peered":"2024-03-23T15:47:28.411310+0000","last_clean":"2024-03-23T15:47:28.411310+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:47:28.411310+0000","last_undegraded":"2024-03-23T15:47:28.411310+0000","last_fullsized":"2024-03-23T15:47:28.411310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682742+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:47:28.682742+0000","last_peered":"2024-03-23T15:47:28.682742+0000","last_clean":"2024-03-23T15:47:28.682742+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:47:28.682742+0000","last_undegraded":"2024-03-23T15:47:28.682742+0000","last_fullsized":"2024-03-23T15:47:28.682742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747920+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:47:28.747920+0000","last_peered":"2024-03-23T15:47:28.747920+0000","last_clean":"2024-03-23T15:47:28.747920+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:47:28.747920+0000","last_undegraded":"2024-03-23T15:47:28.747920+0000","last_fullsized":"2024-03-23T15:47:28.747920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747807+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:47:28.747807+0000","last_peered":"2024-03-23T15:47:28.747807+0000","last_clean":"2024-03-23T15:47:28.747807+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:47:28.747807+0000","last_undegraded":"2024-03-23T15:47:28.747807+0000","last_fullsized":"2024-03-23T15:47:28.747807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411018+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:47:28.411018+0000","last_peered":"2024-03-23T15:47:28.411018+0000","last_clean":"2024-03-23T15:47:28.411018+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:47:28.411018+0000","last_undegraded":"2024-03-23T15:47:28.411018+0000","last_fullsized":"2024-03-23T15:47:28.411018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416777+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:47:28.416777+0000","last_peered":"2024-03-23T15:47:28.416777+0000","last_clean":"2024-03-23T15:47:28.416777+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:47:28.416777+0000","last_undegraded":"2024-03-23T15:47:28.416777+0000","last_fullsized":"2024-03-23T15:47:28.416777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681919+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:47:28.681919+0000","last_peered":"2024-03-23T15:47:28.681919+0000","last_clean":"2024-03-23T15:47:28.681919+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:47:28.681919+0000","last_undegraded":"2024-03-23T15:47:28.681919+0000","last_fullsized":"2024-03-23T15:47:28.681919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682193+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:47:28.682193+0000","last_peered":"2024-03-23T15:47:28.682193+0000","last_clean":"2024-03-23T15:47:28.682193+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:47:28.682193+0000","last_undegraded":"2024-03-23T15:47:28.682193+0000","last_fullsized":"2024-03-23T15:47:28.682193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465791+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:47:28.465791+0000","last_peered":"2024-03-23T15:47:28.465791+0000","last_clean":"2024-03-23T15:47:28.465791+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:47:28.465791+0000","last_undegraded":"2024-03-23T15:47:28.465791+0000","last_fullsized":"2024-03-23T15:47:28.465791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.582464+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:47:33.582464+0000","last_peered":"2024-03-23T15:47:33.582464+0000","last_clean":"2024-03-23T15:47:33.582464+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:47:33.582464+0000","last_undegraded":"2024-03-23T15:47:33.582464+0000","last_fullsized":"2024-03-23T15:47:33.582464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680247+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:47:28.680247+0000","last_peered":"2024-03-23T15:47:28.680247+0000","last_clean":"2024-03-23T15:47:28.680247+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:47:28.680247+0000","last_undegraded":"2024-03-23T15:47:28.680247+0000","last_fullsized":"2024-03-23T15:47:28.680247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.080694+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:47:38.080694+0000","last_peered":"2024-03-23T15:47:38.080694+0000","last_clean":"2024-03-23T15:47:38.080694+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:47:38.080694+0000","last_undegraded":"2024-03-23T15:47:38.080694+0000","last_fullsized":"2024-03-23T15:47:38.080694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.699873+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:47:35.699873+0000","last_peered":"2024-03-23T15:47:35.699873+0000","last_clean":"2024-03-23T15:47:35.699873+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:47:35.699873+0000","last_undegraded":"2024-03-23T15:47:35.699873+0000","last_fullsized":"2024-03-23T15:47:35.699873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411290+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:47:28.411290+0000","last_peered":"2024-03-23T15:47:28.411290+0000","last_clean":"2024-03-23T15:47:28.411290+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:47:28.411290+0000","last_undegraded":"2024-03-23T15:47:28.411290+0000","last_fullsized":"2024-03-23T15:47:28.411290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081264+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:47:38.081264+0000","last_peered":"2024-03-23T15:47:38.081264+0000","last_clean":"2024-03-23T15:47:38.081264+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:47:38.081264+0000","last_undegraded":"2024-03-23T15:47:38.081264+0000","last_fullsized":"2024-03-23T15:47:38.081264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679735+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:47:28.679735+0000","last_peered":"2024-03-23T15:47:28.679735+0000","last_clean":"2024-03-23T15:47:28.679735+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:47:28.679735+0000","last_undegraded":"2024-03-23T15:47:28.679735+0000","last_fullsized":"2024-03-23T15:47:28.679735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.590824+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:47:33.590824+0000","last_peered":"2024-03-23T15:47:33.590824+0000","last_clean":"2024-03-23T15:47:33.590824+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:47:33.590824+0000","last_undegraded":"2024-03-23T15:47:33.590824+0000","last_fullsized":"2024-03-23T15:47:33.590824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682790+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:47:28.682790+0000","last_peered":"2024-03-23T15:47:28.682790+0000","last_clean":"2024-03-23T15:47:28.682790+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:47:28.682790+0000","last_undegraded":"2024-03-23T15:47:28.682790+0000","last_fullsized":"2024-03-23T15:47:28.682790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.902984+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:47:39.902984+0000","last_peered":"2024-03-23T15:47:39.902984+0000","last_clean":"2024-03-23T15:47:39.902984+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:47:39.902984+0000","last_undegraded":"2024-03-23T15:47:39.902984+0000","last_fullsized":"2024-03-23T15:47:39.902984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411427+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:47:28.411427+0000","last_peered":"2024-03-23T15:47:28.411427+0000","last_clean":"2024-03-23T15:47:28.411427+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:47:28.411427+0000","last_undegraded":"2024-03-23T15:47:28.411427+0000","last_fullsized":"2024-03-23T15:47:28.411427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.487354+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:47:33.487354+0000","last_peered":"2024-03-23T15:47:33.487354+0000","last_clean":"2024-03-23T15:47:33.487354+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:47:33.487354+0000","last_undegraded":"2024-03-23T15:47:33.487354+0000","last_fullsized":"2024-03-23T15:47:33.487354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416762+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:47:28.416762+0000","last_peered":"2024-03-23T15:47:28.416762+0000","last_clean":"2024-03-23T15:47:28.416762+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:47:28.416762+0000","last_undegraded":"2024-03-23T15:47:28.416762+0000","last_fullsized":"2024-03-23T15:47:28.416762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:40.029883+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:47:40.029883+0000","last_peered":"2024-03-23T15:47:40.029883+0000","last_clean":"2024-03-23T15:47:40.029883+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:47:40.029883+0000","last_undegraded":"2024-03-23T15:47:40.029883+0000","last_fullsized":"2024-03-23T15:47:40.029883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686519+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:47:28.686519+0000","last_peered":"2024-03-23T15:47:28.686519+0000","last_clean":"2024-03-23T15:47:28.686519+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:47:28.686519+0000","last_undegraded":"2024-03-23T15:47:28.686519+0000","last_fullsized":"2024-03-23T15:47:28.686519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:40.028888+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:47:40.028888+0000","last_peered":"2024-03-23T15:47:40.028888+0000","last_clean":"2024-03-23T15:47:40.028888+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:47:40.028888+0000","last_undegraded":"2024-03-23T15:47:40.028888+0000","last_fullsized":"2024-03-23T15:47:40.028888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747624+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:47:28.747624+0000","last_peered":"2024-03-23T15:47:28.747624+0000","last_clean":"2024-03-23T15:47:28.747624+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:47:28.747624+0000","last_undegraded":"2024-03-23T15:47:28.747624+0000","last_fullsized":"2024-03-23T15:47:28.747624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081081+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:47:38.081081+0000","last_peered":"2024-03-23T15:47:38.081081+0000","last_clean":"2024-03-23T15:47:38.081081+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:47:38.081081+0000","last_undegraded":"2024-03-23T15:47:38.081081+0000","last_fullsized":"2024-03-23T15:47:38.081081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416928+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:47:28.416928+0000","last_peered":"2024-03-23T15:47:28.416928+0000","last_clean":"2024-03-23T15:47:28.416928+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:47:28.416928+0000","last_undegraded":"2024-03-23T15:47:28.416928+0000","last_fullsized":"2024-03-23T15:47:28.416928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.082528+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:47:38.082528+0000","last_peered":"2024-03-23T15:47:38.082528+0000","last_clean":"2024-03-23T15:47:38.082528+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:47:38.082528+0000","last_undegraded":"2024-03-23T15:47:38.082528+0000","last_fullsized":"2024-03-23T15:47:38.082528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465425+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:47:28.465425+0000","last_peered":"2024-03-23T15:47:28.465425+0000","last_clean":"2024-03-23T15:47:28.465425+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:47:28.465425+0000","last_undegraded":"2024-03-23T15:47:28.465425+0000","last_fullsized":"2024-03-23T15:47:28.465425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.076294+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:47:38.076294+0000","last_peered":"2024-03-23T15:47:38.076294+0000","last_clean":"2024-03-23T15:47:38.076294+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:47:38.076294+0000","last_undegraded":"2024-03-23T15:47:38.076294+0000","last_fullsized":"2024-03-23T15:47:38.076294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416623+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:47:28.416623+0000","last_peered":"2024-03-23T15:47:28.416623+0000","last_clean":"2024-03-23T15:47:28.416623+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:47:28.416623+0000","last_undegraded":"2024-03-23T15:47:28.416623+0000","last_fullsized":"2024-03-23T15:47:28.416623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686821+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:47:28.686821+0000","last_peered":"2024-03-23T15:47:28.686821+0000","last_clean":"2024-03-23T15:47:28.686821+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:47:28.686821+0000","last_undegraded":"2024-03-23T15:47:28.686821+0000","last_fullsized":"2024-03-23T15:47:28.686821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682875+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:47:28.682875+0000","last_peered":"2024-03-23T15:47:28.682875+0000","last_clean":"2024-03-23T15:47:28.682875+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:47:28.682875+0000","last_undegraded":"2024-03-23T15:47:28.682875+0000","last_fullsized":"2024-03-23T15:47:28.682875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.901937+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:47:39.901937+0000","last_peered":"2024-03-23T15:47:39.901937+0000","last_clean":"2024-03-23T15:47:39.901937+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:47:39.901937+0000","last_undegraded":"2024-03-23T15:47:39.901937+0000","last_fullsized":"2024-03-23T15:47:39.901937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686676+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:47:28.686676+0000","last_peered":"2024-03-23T15:47:28.686676+0000","last_clean":"2024-03-23T15:47:28.686676+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:47:28.686676+0000","last_undegraded":"2024-03-23T15:47:28.686676+0000","last_fullsized":"2024-03-23T15:47:28.686676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081174+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:47:38.081174+0000","last_peered":"2024-03-23T15:47:38.081174+0000","last_clean":"2024-03-23T15:47:38.081174+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:47:38.081174+0000","last_undegraded":"2024-03-23T15:47:38.081174+0000","last_fullsized":"2024-03-23T15:47:38.081174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686633+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:47:28.686633+0000","last_peered":"2024-03-23T15:47:28.686633+0000","last_clean":"2024-03-23T15:47:28.686633+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:47:28.686633+0000","last_undegraded":"2024-03-23T15:47:28.686633+0000","last_fullsized":"2024-03-23T15:47:28.686633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.082792+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:47:38.082792+0000","last_peered":"2024-03-23T15:47:38.082792+0000","last_clean":"2024-03-23T15:47:38.082792+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:47:38.082792+0000","last_undegraded":"2024-03-23T15:47:38.082792+0000","last_fullsized":"2024-03-23T15:47:38.082792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417319+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:47:28.417319+0000","last_peered":"2024-03-23T15:47:28.417319+0000","last_clean":"2024-03-23T15:47:28.417319+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:47:28.417319+0000","last_undegraded":"2024-03-23T15:47:28.417319+0000","last_fullsized":"2024-03-23T15:47:28.417319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":171,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465446+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:47:28.465446+0000","last_peered":"2024-03-23T15:47:28.465446+0000","last_clean":"2024-03-23T15:47:28.465446+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:47:28.465446+0000","last_undegraded":"2024-03-23T15:47:28.465446+0000","last_fullsized":"2024-03-23T15:47:28.465446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.080765+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:47:38.080765+0000","last_peered":"2024-03-23T15:47:38.080765+0000","last_clean":"2024-03-23T15:47:38.080765+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:47:38.080765+0000","last_undegraded":"2024-03-23T15:47:38.080765+0000","last_fullsized":"2024-03-23T15:47:38.080765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679960+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:47:28.679960+0000","last_peered":"2024-03-23T15:47:28.679960+0000","last_clean":"2024-03-23T15:47:28.679960+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:47:28.679960+0000","last_undegraded":"2024-03-23T15:47:28.679960+0000","last_fullsized":"2024-03-23T15:47:28.679960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.700021+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:47:35.700021+0000","last_peered":"2024-03-23T15:47:35.700021+0000","last_clean":"2024-03-23T15:47:35.700021+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:47:35.700021+0000","last_undegraded":"2024-03-23T15:47:35.700021+0000","last_fullsized":"2024-03-23T15:47:35.700021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747588+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:47:28.747588+0000","last_peered":"2024-03-23T15:47:28.747588+0000","last_clean":"2024-03-23T15:47:28.747588+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:47:28.747588+0000","last_undegraded":"2024-03-23T15:47:28.747588+0000","last_fullsized":"2024-03-23T15:47:28.747588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.688587+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:47:35.688587+0000","last_peered":"2024-03-23T15:47:35.688587+0000","last_clean":"2024-03-23T15:47:35.688587+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:47:35.688587+0000","last_undegraded":"2024-03-23T15:47:35.688587+0000","last_fullsized":"2024-03-23T15:47:35.688587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411099+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:47:28.411099+0000","last_peered":"2024-03-23T15:47:28.411099+0000","last_clean":"2024-03-23T15:47:28.411099+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:47:28.411099+0000","last_undegraded":"2024-03-23T15:47:28.411099+0000","last_fullsized":"2024-03-23T15:47:28.411099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.909335+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:47:39.909335+0000","last_peered":"2024-03-23T15:47:39.909335+0000","last_clean":"2024-03-23T15:47:39.909335+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:47:39.909335+0000","last_undegraded":"2024-03-23T15:47:39.909335+0000","last_fullsized":"2024-03-23T15:47:39.909335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686285+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:47:28.686285+0000","last_peered":"2024-03-23T15:47:28.686285+0000","last_clean":"2024-03-23T15:47:28.686285+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:47:28.686285+0000","last_undegraded":"2024-03-23T15:47:28.686285+0000","last_fullsized":"2024-03-23T15:47:28.686285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411108+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:47:28.411108+0000","last_peered":"2024-03-23T15:47:28.411108+0000","last_clean":"2024-03-23T15:47:28.411108+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:47:28.411108+0000","last_undegraded":"2024-03-23T15:47:28.411108+0000","last_fullsized":"2024-03-23T15:47:28.411108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682390+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:47:28.682390+0000","last_peered":"2024-03-23T15:47:28.682390+0000","last_clean":"2024-03-23T15:47:28.682390+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:47:28.682390+0000","last_undegraded":"2024-03-23T15:47:28.682390+0000","last_fullsized":"2024-03-23T15:47:28.682390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081361+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:47:38.081361+0000","last_peered":"2024-03-23T15:47:38.081361+0000","last_clean":"2024-03-23T15:47:38.081361+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:47:38.081361+0000","last_undegraded":"2024-03-23T15:47:38.081361+0000","last_fullsized":"2024-03-23T15:47:38.081361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747808+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:47:28.747808+0000","last_peered":"2024-03-23T15:47:28.747808+0000","last_clean":"2024-03-23T15:47:28.747808+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:47:28.747808+0000","last_undegraded":"2024-03-23T15:47:28.747808+0000","last_fullsized":"2024-03-23T15:47:28.747808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.082970+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:47:38.082970+0000","last_peered":"2024-03-23T15:47:38.082970+0000","last_clean":"2024-03-23T15:47:38.082970+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:47:38.082970+0000","last_undegraded":"2024-03-23T15:47:38.082970+0000","last_fullsized":"2024-03-23T15:47:38.082970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680240+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:47:28.680240+0000","last_peered":"2024-03-23T15:47:28.680240+0000","last_clean":"2024-03-23T15:47:28.680240+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:47:28.680240+0000","last_undegraded":"2024-03-23T15:47:28.680240+0000","last_fullsized":"2024-03-23T15:47:28.680240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.911567+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:47:39.911567+0000","last_peered":"2024-03-23T15:47:39.911567+0000","last_clean":"2024-03-23T15:47:39.911567+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:47:39.911567+0000","last_undegraded":"2024-03-23T15:47:39.911567+0000","last_fullsized":"2024-03-23T15:47:39.911567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680148+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:47:28.680148+0000","last_peered":"2024-03-23T15:47:28.680148+0000","last_clean":"2024-03-23T15:47:28.680148+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:47:28.680148+0000","last_undegraded":"2024-03-23T15:47:28.680148+0000","last_fullsized":"2024-03-23T15:47:28.680148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.602195+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:47:33.602195+0000","last_peered":"2024-03-23T15:47:33.602195+0000","last_clean":"2024-03-23T15:47:33.602195+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:47:33.602195+0000","last_undegraded":"2024-03-23T15:47:33.602195+0000","last_fullsized":"2024-03-23T15:47:33.602195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416764+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:47:28.416764+0000","last_peered":"2024-03-23T15:47:28.416764+0000","last_clean":"2024-03-23T15:47:28.416764+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:47:28.416764+0000","last_undegraded":"2024-03-23T15:47:28.416764+0000","last_fullsized":"2024-03-23T15:47:28.416764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.910489+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:47:39.910489+0000","last_peered":"2024-03-23T15:47:39.910489+0000","last_clean":"2024-03-23T15:47:39.910489+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:47:39.910489+0000","last_undegraded":"2024-03-23T15:47:39.910489+0000","last_fullsized":"2024-03-23T15:47:39.910489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679874+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:47:28.679874+0000","last_peered":"2024-03-23T15:47:28.679874+0000","last_clean":"2024-03-23T15:47:28.679874+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:47:28.679874+0000","last_undegraded":"2024-03-23T15:47:28.679874+0000","last_fullsized":"2024-03-23T15:47:28.679874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.077900+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:47:38.077900+0000","last_peered":"2024-03-23T15:47:38.077900+0000","last_clean":"2024-03-23T15:47:38.077900+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:47:38.077900+0000","last_undegraded":"2024-03-23T15:47:38.077900+0000","last_fullsized":"2024-03-23T15:47:38.077900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682112+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:47:28.682112+0000","last_peered":"2024-03-23T15:47:28.682112+0000","last_clean":"2024-03-23T15:47:28.682112+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:47:28.682112+0000","last_undegraded":"2024-03-23T15:47:28.682112+0000","last_fullsized":"2024-03-23T15:47:28.682112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.493555+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:47:33.493555+0000","last_peered":"2024-03-23T15:47:33.493555+0000","last_clean":"2024-03-23T15:47:33.493555+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:47:33.493555+0000","last_undegraded":"2024-03-23T15:47:33.493555+0000","last_fullsized":"2024-03-23T15:47:33.493555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416913+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:47:28.416913+0000","last_peered":"2024-03-23T15:47:28.416913+0000","last_clean":"2024-03-23T15:47:28.416913+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:47:28.416913+0000","last_undegraded":"2024-03-23T15:47:28.416913+0000","last_fullsized":"2024-03-23T15:47:28.416913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416753+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:47:28.416753+0000","last_peered":"2024-03-23T15:47:28.416753+0000","last_clean":"2024-03-23T15:47:28.416753+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:47:28.416753+0000","last_undegraded":"2024-03-23T15:47:28.416753+0000","last_fullsized":"2024-03-23T15:47:28.416753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.908492+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:47:39.908492+0000","last_peered":"2024-03-23T15:47:39.908492+0000","last_clean":"2024-03-23T15:47:39.908492+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:47:39.908492+0000","last_undegraded":"2024-03-23T15:47:39.908492+0000","last_fullsized":"2024-03-23T15:47:39.908492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465591+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:47:28.465591+0000","last_peered":"2024-03-23T15:47:28.465591+0000","last_clean":"2024-03-23T15:47:28.465591+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:47:28.465591+0000","last_undegraded":"2024-03-23T15:47:28.465591+0000","last_fullsized":"2024-03-23T15:47:28.465591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.907672+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:47:39.907672+0000","last_peered":"2024-03-23T15:47:39.907672+0000","last_clean":"2024-03-23T15:47:39.907672+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:47:39.907672+0000","last_undegraded":"2024-03-23T15:47:39.907672+0000","last_fullsized":"2024-03-23T15:47:39.907672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411323+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:47:28.411323+0000","last_peered":"2024-03-23T15:47:28.411323+0000","last_clean":"2024-03-23T15:47:28.411323+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:47:28.411323+0000","last_undegraded":"2024-03-23T15:47:28.411323+0000","last_fullsized":"2024-03-23T15:47:28.411323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747051+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:47:28.747051+0000","last_peered":"2024-03-23T15:47:28.747051+0000","last_clean":"2024-03-23T15:47:28.747051+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:47:28.747051+0000","last_undegraded":"2024-03-23T15:47:28.747051+0000","last_fullsized":"2024-03-23T15:47:28.747051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411212+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:47:28.411212+0000","last_peered":"2024-03-23T15:47:28.411212+0000","last_clean":"2024-03-23T15:47:28.411212+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:47:28.411212+0000","last_undegraded":"2024-03-23T15:47:28.411212+0000","last_fullsized":"2024-03-23T15:47:28.411212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.910233+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:47:39.910233+0000","last_peered":"2024-03-23T15:47:39.910233+0000","last_clean":"2024-03-23T15:47:39.910233+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:47:39.910233+0000","last_undegraded":"2024-03-23T15:47:39.910233+0000","last_fullsized":"2024-03-23T15:47:39.910233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":77045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":78371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":667185,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":79441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":78713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":667773,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":78533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T15:47:43.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T15:47:44.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.040+0000 7fbc7493e640 1 Processor -- start 2024-03-23T15:47:44.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.040+0000 7fbc7493e640 1 -- start start 2024-03-23T15:47:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.041+0000 7fbc7493e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 0x7fbc7015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.041+0000 7fbc7493e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbc700c6000 0x7fbc700c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.041+0000 7fbc7493e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc700cbcb0 con 0x7fbc700caa00 2024-03-23T15:47:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.041+0000 7fbc7493e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbc700c5be0 con 0x7fbc700c6000 2024-03-23T15:47:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.041+0000 7fbc7493e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbc700c69b0 con 0x7fbc701544b0 2024-03-23T15:47:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.042+0000 7fbc6ed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbc700c6000 0x7fbc700c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.042+0000 7fbc6e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 0x7fbc7015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.042+0000 7fbc6e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 0x7fbc7015a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48904/0 (socket says 172.21.15.129:48904) 2024-03-23T15:47:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.043+0000 7fbc6e59c640 1 -- 172.21.15.129:0/2843958986 learned_addr learned my addr 172.21.15.129:0/2843958986 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.045+0000 7fbc6e59c640 1 -- 172.21.15.129:0/2843958986 >> v2:172.21.15.157:3300/0 conn(0x7fbc700c6000 msgr2=0x7fbc700c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.046+0000 7fbc6e59c640 1 --2- 172.21.15.129:0/2843958986 >> v2:172.21.15.157:3300/0 conn(0x7fbc700c6000 0x7fbc700c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.046+0000 7fbc6e59c640 1 -- 172.21.15.129:0/2843958986 >> v1:172.21.15.129:6789/0 conn(0x7fbc700caa00 legacy=0x7fbc7006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.046+0000 7fbc6e59c640 1 -- 172.21.15.129:0/2843958986 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7015b280 con 0x7fbc701544b0 2024-03-23T15:47:44.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.048+0000 7fbc6e59c640 1 --2- 172.21.15.129:0/2843958986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 0x7fbc7015a2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009870 tx=0x7fbc6002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3455d40e0c532f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.059+0000 7fbc6d59a640 1 -- 172.21.15.129:0/2843958986 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60030840 con 0x7fbc701544b0 2024-03-23T15:47:44.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.139+0000 7fbc6d59a640 1 -- 172.21.15.129:0/2843958986 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc600309a0 con 0x7fbc701544b0 2024-03-23T15:47:44.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.139+0000 7fbc6d59a640 1 -- 172.21.15.129:0/2843958986 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60030d30 con 0x7fbc701544b0 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 -- 172.21.15.129:0/2843958986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 msgr2=0x7fbc7015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 --2- 172.21.15.129:0/2843958986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 0x7fbc7015a2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009870 tx=0x7fbc6002b980 comp rx=0 tx=0).stop 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 -- 172.21.15.129:0/2843958986 shutdown_connections 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 --2- 172.21.15.129:0/2843958986 >> v2:172.21.15.157:3300/0 conn(0x7fbc700c6000 0x7fbc700c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 --2- 172.21.15.129:0/2843958986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc701544b0 0x7fbc7015a2e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 -- 172.21.15.129:0/2843958986 >> 172.21.15.129:0/2843958986 conn(0x7fbc70061b50 msgr2=0x7fbc70061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 -- 172.21.15.129:0/2843958986 shutdown_connections 2024-03-23T15:47:44.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.140+0000 7fbc7493e640 1 -- 172.21.15.129:0/2843958986 wait complete. 2024-03-23T15:47:44.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 Processor -- start 2024-03-23T15:47:44.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 -- start start 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 0x7fbc70156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc700caa00 0x7fbc701484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc701544b0 0x7fbc70154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701f4240 con 0x7fbc700c6000 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70157a60 con 0x7fbc701544b0 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.142+0000 7fbc7493e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbc70157780 con 0x7fbc700caa00 2024-03-23T15:47:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.143+0000 7fbc6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 0x7fbc70156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.143+0000 7fbc6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 0x7fbc70156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60746/0 (socket says 172.21.15.129:60746) 2024-03-23T15:47:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.143+0000 7fbc6e59c640 1 -- 172.21.15.129:0/4087882670 learned_addr learned my addr 172.21.15.129:0/4087882670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.143+0000 7fbc6ed9d640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc701544b0 0x7fbc70154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.146+0000 7fbc6e59c640 1 -- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc700caa00 msgr2=0x7fbc701484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.146+0000 7fbc6e59c640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc700caa00 0x7fbc701484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.146+0000 7fbc6e59c640 1 -- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc701544b0 msgr2=0x7fbc70154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.146+0000 7fbc6e59c640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc701544b0 0x7fbc70154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.146+0000 7fbc6e59c640 1 -- 172.21.15.129:0/4087882670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc600094d0 con 0x7fbc700c6000 2024-03-23T15:47:44.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.149+0000 7fbc6e59c640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 0x7fbc70156580 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc600090b0 tx=0x7fbc600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:44.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.151+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6003e070 con 0x7fbc700c6000 2024-03-23T15:47:44.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.151+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc701f3910 con 0x7fbc700c6000 2024-03-23T15:47:44.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.151+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc701f3ed0 con 0x7fbc700c6000 2024-03-23T15:47:44.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.152+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc70075740 con 0x7fbc700c6000 2024-03-23T15:47:44.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.358+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc60039800 con 0x7fbc700c6000 2024-03-23T15:47:44.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.421+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60039b70 con 0x7fbc700c6000 2024-03-23T15:47:44.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.422+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc60042070 con 0x7fbc700c6000 2024-03-23T15:47:44.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.423+0000 7fbc5effd640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc5006b810 0x7fbc5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.423+0000 7fbc6dd9b640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc5006b810 0x7fbc5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.423+0000 7fbc6dd9b640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc5006b810 0x7fbc5006dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbc700c7370 tx=0x7fbc64007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:44.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.436+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbc6007ada0 con 0x7fbc700c6000 2024-03-23T15:47:44.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.437+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc60077040 con 0x7fbc700c6000 2024-03-23T15:47:44.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.639+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc7015afa0 con 0x7fbc5006b810 2024-03-23T15:47:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.646+0000 7fbc5effd640 1 -- 172.21.15.129:0/4087882670 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252041 (secure 0 0 0) 0x7fbc640050d0 con 0x7fbc5006b810 2024-03-23T15:47:44.651 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:44.652 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T15:47:44.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.648+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc5006b810 msgr2=0x7fbc5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.648+0000 7fbc7493e640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc5006b810 0x7fbc5006dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbc700c7370 tx=0x7fbc64007990 comp rx=0 tx=0).stop 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.648+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 msgr2=0x7fbc70156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.648+0000 7fbc7493e640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 0x7fbc70156580 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc600090b0 tx=0x7fbc600046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 shutdown_connections 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc5006b810 0x7fbc5006dcd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc701544b0 0x7fbc70154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc700caa00 0x7fbc701484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 --2- 172.21.15.129:0/4087882670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc700c6000 0x7fbc70156580 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 >> 172.21.15.129:0/4087882670 conn(0x7fbc70061b50 msgr2=0x7fbc700b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 shutdown_connections 2024-03-23T15:47:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.649+0000 7fbc7493e640 1 -- 172.21.15.129:0/4087882670 wait complete. 2024-03-23T15:47:44.664 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-23T15:47:44.409364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":362,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230688,"kb_used_data":2080,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754744032,"statfs":{"total":773094113280,"available":772857888768,"internally_reserved":0,"allocated":2129920,"data_stored":1804732,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5624,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":158,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002755"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686665+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:47:28.686665+0000","last_peered":"2024-03-23T15:47:28.686665+0000","last_clean":"2024-03-23T15:47:28.686665+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:47:28.686665+0000","last_undegraded":"2024-03-23T15:47:28.686665+0000","last_fullsized":"2024-03-23T15:47:28.686665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411203+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:47:28.411203+0000","last_peered":"2024-03-23T15:47:28.411203+0000","last_clean":"2024-03-23T15:47:28.411203+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:47:28.411203+0000","last_undegraded":"2024-03-23T15:47:28.411203+0000","last_fullsized":"2024-03-23T15:47:28.411203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411144+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:47:28.411144+0000","last_peered":"2024-03-23T15:47:28.411144+0000","last_clean":"2024-03-23T15:47:28.411144+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:47:28.411144+0000","last_undegraded":"2024-03-23T15:47:28.411144+0000","last_fullsized":"2024-03-23T15:47:28.411144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682515+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:47:28.682515+0000","last_peered":"2024-03-23T15:47:28.682515+0000","last_clean":"2024-03-23T15:47:28.682515+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:47:28.682515+0000","last_undegraded":"2024-03-23T15:47:28.682515+0000","last_fullsized":"2024-03-23T15:47:28.682515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686424+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:47:28.686424+0000","last_peered":"2024-03-23T15:47:28.686424+0000","last_clean":"2024-03-23T15:47:28.686424+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:47:28.686424+0000","last_undegraded":"2024-03-23T15:47:28.686424+0000","last_fullsized":"2024-03-23T15:47:28.686424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682361+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:47:28.682361+0000","last_peered":"2024-03-23T15:47:28.682361+0000","last_clean":"2024-03-23T15:47:28.682361+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:47:28.682361+0000","last_undegraded":"2024-03-23T15:47:28.682361+0000","last_fullsized":"2024-03-23T15:47:28.682361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686125+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:47:28.686125+0000","last_peered":"2024-03-23T15:47:28.686125+0000","last_clean":"2024-03-23T15:47:28.686125+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:47:28.686125+0000","last_undegraded":"2024-03-23T15:47:28.686125+0000","last_fullsized":"2024-03-23T15:47:28.686125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410933+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:47:28.410933+0000","last_peered":"2024-03-23T15:47:28.410933+0000","last_clean":"2024-03-23T15:47:28.410933+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:47:28.410933+0000","last_undegraded":"2024-03-23T15:47:28.410933+0000","last_fullsized":"2024-03-23T15:47:28.410933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465785+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:47:28.465785+0000","last_peered":"2024-03-23T15:47:28.465785+0000","last_clean":"2024-03-23T15:47:28.465785+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:47:28.465785+0000","last_undegraded":"2024-03-23T15:47:28.465785+0000","last_fullsized":"2024-03-23T15:47:28.465785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686368+0000","last_change":"2024-03-23T15:47:27.453034+0000","last_active":"2024-03-23T15:47:28.686368+0000","last_peered":"2024-03-23T15:47:28.686368+0000","last_clean":"2024-03-23T15:47:28.686368+0000","last_became_active":"2024-03-23T15:47:27.452786+0000","last_became_peered":"2024-03-23T15:47:27.452786+0000","last_unstale":"2024-03-23T15:47:28.686368+0000","last_undegraded":"2024-03-23T15:47:28.686368+0000","last_fullsized":"2024-03-23T15:47:28.686368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686412+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:47:28.686412+0000","last_peered":"2024-03-23T15:47:28.686412+0000","last_clean":"2024-03-23T15:47:28.686412+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:47:28.686412+0000","last_undegraded":"2024-03-23T15:47:28.686412+0000","last_fullsized":"2024-03-23T15:47:28.686412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747756+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:47:28.747756+0000","last_peered":"2024-03-23T15:47:28.747756+0000","last_clean":"2024-03-23T15:47:28.747756+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:47:28.747756+0000","last_undegraded":"2024-03-23T15:47:28.747756+0000","last_fullsized":"2024-03-23T15:47:28.747756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747609+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:47:28.747609+0000","last_peered":"2024-03-23T15:47:28.747609+0000","last_clean":"2024-03-23T15:47:28.747609+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:47:28.747609+0000","last_undegraded":"2024-03-23T15:47:28.747609+0000","last_fullsized":"2024-03-23T15:47:28.747609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410854+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:47:28.410854+0000","last_peered":"2024-03-23T15:47:28.410854+0000","last_clean":"2024-03-23T15:47:28.410854+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:47:28.410854+0000","last_undegraded":"2024-03-23T15:47:28.410854+0000","last_fullsized":"2024-03-23T15:47:28.410854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416608+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:47:28.416608+0000","last_peered":"2024-03-23T15:47:28.416608+0000","last_clean":"2024-03-23T15:47:28.416608+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:47:28.416608+0000","last_undegraded":"2024-03-23T15:47:28.416608+0000","last_fullsized":"2024-03-23T15:47:28.416608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417524+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:47:28.417524+0000","last_peered":"2024-03-23T15:47:28.417524+0000","last_clean":"2024-03-23T15:47:28.417524+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:47:28.417524+0000","last_undegraded":"2024-03-23T15:47:28.417524+0000","last_fullsized":"2024-03-23T15:47:28.417524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681899+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:47:28.681899+0000","last_peered":"2024-03-23T15:47:28.681899+0000","last_clean":"2024-03-23T15:47:28.681899+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:47:28.681899+0000","last_undegraded":"2024-03-23T15:47:28.681899+0000","last_fullsized":"2024-03-23T15:47:28.681899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417419+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:47:28.417419+0000","last_peered":"2024-03-23T15:47:28.417419+0000","last_clean":"2024-03-23T15:47:28.417419+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:47:28.417419+0000","last_undegraded":"2024-03-23T15:47:28.417419+0000","last_fullsized":"2024-03-23T15:47:28.417419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686539+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:47:28.686539+0000","last_peered":"2024-03-23T15:47:28.686539+0000","last_clean":"2024-03-23T15:47:28.686539+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:47:28.686539+0000","last_undegraded":"2024-03-23T15:47:28.686539+0000","last_fullsized":"2024-03-23T15:47:28.686539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679986+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:47:28.679986+0000","last_peered":"2024-03-23T15:47:28.679986+0000","last_clean":"2024-03-23T15:47:28.679986+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:47:28.679986+0000","last_undegraded":"2024-03-23T15:47:28.679986+0000","last_fullsized":"2024-03-23T15:47:28.679986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465648+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:47:28.465648+0000","last_peered":"2024-03-23T15:47:28.465648+0000","last_clean":"2024-03-23T15:47:28.465648+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:47:28.465648+0000","last_undegraded":"2024-03-23T15:47:28.465648+0000","last_fullsized":"2024-03-23T15:47:28.465648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.410906+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:47:28.410906+0000","last_peered":"2024-03-23T15:47:28.410906+0000","last_clean":"2024-03-23T15:47:28.410906+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:47:28.410906+0000","last_undegraded":"2024-03-23T15:47:28.410906+0000","last_fullsized":"2024-03-23T15:47:28.410906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416609+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:47:28.416609+0000","last_peered":"2024-03-23T15:47:28.416609+0000","last_clean":"2024-03-23T15:47:28.416609+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:47:28.416609+0000","last_undegraded":"2024-03-23T15:47:28.416609+0000","last_fullsized":"2024-03-23T15:47:28.416609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411310+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:47:28.411310+0000","last_peered":"2024-03-23T15:47:28.411310+0000","last_clean":"2024-03-23T15:47:28.411310+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:47:28.411310+0000","last_undegraded":"2024-03-23T15:47:28.411310+0000","last_fullsized":"2024-03-23T15:47:28.411310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682742+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:47:28.682742+0000","last_peered":"2024-03-23T15:47:28.682742+0000","last_clean":"2024-03-23T15:47:28.682742+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:47:28.682742+0000","last_undegraded":"2024-03-23T15:47:28.682742+0000","last_fullsized":"2024-03-23T15:47:28.682742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747920+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:47:28.747920+0000","last_peered":"2024-03-23T15:47:28.747920+0000","last_clean":"2024-03-23T15:47:28.747920+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:47:28.747920+0000","last_undegraded":"2024-03-23T15:47:28.747920+0000","last_fullsized":"2024-03-23T15:47:28.747920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747807+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:47:28.747807+0000","last_peered":"2024-03-23T15:47:28.747807+0000","last_clean":"2024-03-23T15:47:28.747807+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:47:28.747807+0000","last_undegraded":"2024-03-23T15:47:28.747807+0000","last_fullsized":"2024-03-23T15:47:28.747807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411018+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:47:28.411018+0000","last_peered":"2024-03-23T15:47:28.411018+0000","last_clean":"2024-03-23T15:47:28.411018+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:47:28.411018+0000","last_undegraded":"2024-03-23T15:47:28.411018+0000","last_fullsized":"2024-03-23T15:47:28.411018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416777+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:47:28.416777+0000","last_peered":"2024-03-23T15:47:28.416777+0000","last_clean":"2024-03-23T15:47:28.416777+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:47:28.416777+0000","last_undegraded":"2024-03-23T15:47:28.416777+0000","last_fullsized":"2024-03-23T15:47:28.416777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.681919+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:47:28.681919+0000","last_peered":"2024-03-23T15:47:28.681919+0000","last_clean":"2024-03-23T15:47:28.681919+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:47:28.681919+0000","last_undegraded":"2024-03-23T15:47:28.681919+0000","last_fullsized":"2024-03-23T15:47:28.681919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682193+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:47:28.682193+0000","last_peered":"2024-03-23T15:47:28.682193+0000","last_clean":"2024-03-23T15:47:28.682193+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:47:28.682193+0000","last_undegraded":"2024-03-23T15:47:28.682193+0000","last_fullsized":"2024-03-23T15:47:28.682193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465791+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:47:28.465791+0000","last_peered":"2024-03-23T15:47:28.465791+0000","last_clean":"2024-03-23T15:47:28.465791+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:47:28.465791+0000","last_undegraded":"2024-03-23T15:47:28.465791+0000","last_fullsized":"2024-03-23T15:47:28.465791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.582464+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:47:33.582464+0000","last_peered":"2024-03-23T15:47:33.582464+0000","last_clean":"2024-03-23T15:47:33.582464+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:47:33.582464+0000","last_undegraded":"2024-03-23T15:47:33.582464+0000","last_fullsized":"2024-03-23T15:47:33.582464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680247+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:47:28.680247+0000","last_peered":"2024-03-23T15:47:28.680247+0000","last_clean":"2024-03-23T15:47:28.680247+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:47:28.680247+0000","last_undegraded":"2024-03-23T15:47:28.680247+0000","last_fullsized":"2024-03-23T15:47:28.680247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.080694+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:47:38.080694+0000","last_peered":"2024-03-23T15:47:38.080694+0000","last_clean":"2024-03-23T15:47:38.080694+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:47:38.080694+0000","last_undegraded":"2024-03-23T15:47:38.080694+0000","last_fullsized":"2024-03-23T15:47:38.080694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.699873+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:47:35.699873+0000","last_peered":"2024-03-23T15:47:35.699873+0000","last_clean":"2024-03-23T15:47:35.699873+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:47:35.699873+0000","last_undegraded":"2024-03-23T15:47:35.699873+0000","last_fullsized":"2024-03-23T15:47:35.699873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411290+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:47:28.411290+0000","last_peered":"2024-03-23T15:47:28.411290+0000","last_clean":"2024-03-23T15:47:28.411290+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:47:28.411290+0000","last_undegraded":"2024-03-23T15:47:28.411290+0000","last_fullsized":"2024-03-23T15:47:28.411290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081264+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:47:38.081264+0000","last_peered":"2024-03-23T15:47:38.081264+0000","last_clean":"2024-03-23T15:47:38.081264+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:47:38.081264+0000","last_undegraded":"2024-03-23T15:47:38.081264+0000","last_fullsized":"2024-03-23T15:47:38.081264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679735+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:47:28.679735+0000","last_peered":"2024-03-23T15:47:28.679735+0000","last_clean":"2024-03-23T15:47:28.679735+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:47:28.679735+0000","last_undegraded":"2024-03-23T15:47:28.679735+0000","last_fullsized":"2024-03-23T15:47:28.679735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.590824+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:47:33.590824+0000","last_peered":"2024-03-23T15:47:33.590824+0000","last_clean":"2024-03-23T15:47:33.590824+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:47:33.590824+0000","last_undegraded":"2024-03-23T15:47:33.590824+0000","last_fullsized":"2024-03-23T15:47:33.590824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682790+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:47:28.682790+0000","last_peered":"2024-03-23T15:47:28.682790+0000","last_clean":"2024-03-23T15:47:28.682790+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:47:28.682790+0000","last_undegraded":"2024-03-23T15:47:28.682790+0000","last_fullsized":"2024-03-23T15:47:28.682790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.902984+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:47:39.902984+0000","last_peered":"2024-03-23T15:47:39.902984+0000","last_clean":"2024-03-23T15:47:39.902984+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:47:39.902984+0000","last_undegraded":"2024-03-23T15:47:39.902984+0000","last_fullsized":"2024-03-23T15:47:39.902984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411427+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:47:28.411427+0000","last_peered":"2024-03-23T15:47:28.411427+0000","last_clean":"2024-03-23T15:47:28.411427+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:47:28.411427+0000","last_undegraded":"2024-03-23T15:47:28.411427+0000","last_fullsized":"2024-03-23T15:47:28.411427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.487354+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:47:33.487354+0000","last_peered":"2024-03-23T15:47:33.487354+0000","last_clean":"2024-03-23T15:47:33.487354+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:47:33.487354+0000","last_undegraded":"2024-03-23T15:47:33.487354+0000","last_fullsized":"2024-03-23T15:47:33.487354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416762+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:47:28.416762+0000","last_peered":"2024-03-23T15:47:28.416762+0000","last_clean":"2024-03-23T15:47:28.416762+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:47:28.416762+0000","last_undegraded":"2024-03-23T15:47:28.416762+0000","last_fullsized":"2024-03-23T15:47:28.416762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:40.029883+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:47:40.029883+0000","last_peered":"2024-03-23T15:47:40.029883+0000","last_clean":"2024-03-23T15:47:40.029883+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:47:40.029883+0000","last_undegraded":"2024-03-23T15:47:40.029883+0000","last_fullsized":"2024-03-23T15:47:40.029883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686519+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:47:28.686519+0000","last_peered":"2024-03-23T15:47:28.686519+0000","last_clean":"2024-03-23T15:47:28.686519+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:47:28.686519+0000","last_undegraded":"2024-03-23T15:47:28.686519+0000","last_fullsized":"2024-03-23T15:47:28.686519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:40.028888+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:47:40.028888+0000","last_peered":"2024-03-23T15:47:40.028888+0000","last_clean":"2024-03-23T15:47:40.028888+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:47:40.028888+0000","last_undegraded":"2024-03-23T15:47:40.028888+0000","last_fullsized":"2024-03-23T15:47:40.028888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747624+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:47:28.747624+0000","last_peered":"2024-03-23T15:47:28.747624+0000","last_clean":"2024-03-23T15:47:28.747624+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:47:28.747624+0000","last_undegraded":"2024-03-23T15:47:28.747624+0000","last_fullsized":"2024-03-23T15:47:28.747624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081081+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:47:38.081081+0000","last_peered":"2024-03-23T15:47:38.081081+0000","last_clean":"2024-03-23T15:47:38.081081+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:47:38.081081+0000","last_undegraded":"2024-03-23T15:47:38.081081+0000","last_fullsized":"2024-03-23T15:47:38.081081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416928+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:47:28.416928+0000","last_peered":"2024-03-23T15:47:28.416928+0000","last_clean":"2024-03-23T15:47:28.416928+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:47:28.416928+0000","last_undegraded":"2024-03-23T15:47:28.416928+0000","last_fullsized":"2024-03-23T15:47:28.416928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.082528+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:47:38.082528+0000","last_peered":"2024-03-23T15:47:38.082528+0000","last_clean":"2024-03-23T15:47:38.082528+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:47:38.082528+0000","last_undegraded":"2024-03-23T15:47:38.082528+0000","last_fullsized":"2024-03-23T15:47:38.082528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465425+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:47:28.465425+0000","last_peered":"2024-03-23T15:47:28.465425+0000","last_clean":"2024-03-23T15:47:28.465425+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:47:28.465425+0000","last_undegraded":"2024-03-23T15:47:28.465425+0000","last_fullsized":"2024-03-23T15:47:28.465425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.076294+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:47:38.076294+0000","last_peered":"2024-03-23T15:47:38.076294+0000","last_clean":"2024-03-23T15:47:38.076294+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:47:38.076294+0000","last_undegraded":"2024-03-23T15:47:38.076294+0000","last_fullsized":"2024-03-23T15:47:38.076294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416623+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:47:28.416623+0000","last_peered":"2024-03-23T15:47:28.416623+0000","last_clean":"2024-03-23T15:47:28.416623+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:47:28.416623+0000","last_undegraded":"2024-03-23T15:47:28.416623+0000","last_fullsized":"2024-03-23T15:47:28.416623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686821+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:47:28.686821+0000","last_peered":"2024-03-23T15:47:28.686821+0000","last_clean":"2024-03-23T15:47:28.686821+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:47:28.686821+0000","last_undegraded":"2024-03-23T15:47:28.686821+0000","last_fullsized":"2024-03-23T15:47:28.686821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682875+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:47:28.682875+0000","last_peered":"2024-03-23T15:47:28.682875+0000","last_clean":"2024-03-23T15:47:28.682875+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:47:28.682875+0000","last_undegraded":"2024-03-23T15:47:28.682875+0000","last_fullsized":"2024-03-23T15:47:28.682875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.901937+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:47:39.901937+0000","last_peered":"2024-03-23T15:47:39.901937+0000","last_clean":"2024-03-23T15:47:39.901937+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:47:39.901937+0000","last_undegraded":"2024-03-23T15:47:39.901937+0000","last_fullsized":"2024-03-23T15:47:39.901937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686676+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:47:28.686676+0000","last_peered":"2024-03-23T15:47:28.686676+0000","last_clean":"2024-03-23T15:47:28.686676+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:47:28.686676+0000","last_undegraded":"2024-03-23T15:47:28.686676+0000","last_fullsized":"2024-03-23T15:47:28.686676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081174+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:47:38.081174+0000","last_peered":"2024-03-23T15:47:38.081174+0000","last_clean":"2024-03-23T15:47:38.081174+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:47:38.081174+0000","last_undegraded":"2024-03-23T15:47:38.081174+0000","last_fullsized":"2024-03-23T15:47:38.081174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686633+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:47:28.686633+0000","last_peered":"2024-03-23T15:47:28.686633+0000","last_clean":"2024-03-23T15:47:28.686633+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:47:28.686633+0000","last_undegraded":"2024-03-23T15:47:28.686633+0000","last_fullsized":"2024-03-23T15:47:28.686633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.082792+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:47:38.082792+0000","last_peered":"2024-03-23T15:47:38.082792+0000","last_clean":"2024-03-23T15:47:38.082792+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:47:38.082792+0000","last_undegraded":"2024-03-23T15:47:38.082792+0000","last_fullsized":"2024-03-23T15:47:38.082792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.417319+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:47:28.417319+0000","last_peered":"2024-03-23T15:47:28.417319+0000","last_clean":"2024-03-23T15:47:28.417319+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:47:28.417319+0000","last_undegraded":"2024-03-23T15:47:28.417319+0000","last_fullsized":"2024-03-23T15:47:28.417319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":171,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465446+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:47:28.465446+0000","last_peered":"2024-03-23T15:47:28.465446+0000","last_clean":"2024-03-23T15:47:28.465446+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:47:28.465446+0000","last_undegraded":"2024-03-23T15:47:28.465446+0000","last_fullsized":"2024-03-23T15:47:28.465446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.080765+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:47:38.080765+0000","last_peered":"2024-03-23T15:47:38.080765+0000","last_clean":"2024-03-23T15:47:38.080765+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:47:38.080765+0000","last_undegraded":"2024-03-23T15:47:38.080765+0000","last_fullsized":"2024-03-23T15:47:38.080765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679960+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:47:28.679960+0000","last_peered":"2024-03-23T15:47:28.679960+0000","last_clean":"2024-03-23T15:47:28.679960+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:47:28.679960+0000","last_undegraded":"2024-03-23T15:47:28.679960+0000","last_fullsized":"2024-03-23T15:47:28.679960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.700021+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:47:35.700021+0000","last_peered":"2024-03-23T15:47:35.700021+0000","last_clean":"2024-03-23T15:47:35.700021+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:47:35.700021+0000","last_undegraded":"2024-03-23T15:47:35.700021+0000","last_fullsized":"2024-03-23T15:47:35.700021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747588+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:47:28.747588+0000","last_peered":"2024-03-23T15:47:28.747588+0000","last_clean":"2024-03-23T15:47:28.747588+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:47:28.747588+0000","last_undegraded":"2024-03-23T15:47:28.747588+0000","last_fullsized":"2024-03-23T15:47:28.747588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:35.688587+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:47:35.688587+0000","last_peered":"2024-03-23T15:47:35.688587+0000","last_clean":"2024-03-23T15:47:35.688587+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:47:35.688587+0000","last_undegraded":"2024-03-23T15:47:35.688587+0000","last_fullsized":"2024-03-23T15:47:35.688587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411099+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:47:28.411099+0000","last_peered":"2024-03-23T15:47:28.411099+0000","last_clean":"2024-03-23T15:47:28.411099+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:47:28.411099+0000","last_undegraded":"2024-03-23T15:47:28.411099+0000","last_fullsized":"2024-03-23T15:47:28.411099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.909335+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:47:39.909335+0000","last_peered":"2024-03-23T15:47:39.909335+0000","last_clean":"2024-03-23T15:47:39.909335+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:47:39.909335+0000","last_undegraded":"2024-03-23T15:47:39.909335+0000","last_fullsized":"2024-03-23T15:47:39.909335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.686285+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:47:28.686285+0000","last_peered":"2024-03-23T15:47:28.686285+0000","last_clean":"2024-03-23T15:47:28.686285+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:47:28.686285+0000","last_undegraded":"2024-03-23T15:47:28.686285+0000","last_fullsized":"2024-03-23T15:47:28.686285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411108+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:47:28.411108+0000","last_peered":"2024-03-23T15:47:28.411108+0000","last_clean":"2024-03-23T15:47:28.411108+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:47:28.411108+0000","last_undegraded":"2024-03-23T15:47:28.411108+0000","last_fullsized":"2024-03-23T15:47:28.411108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682390+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:47:28.682390+0000","last_peered":"2024-03-23T15:47:28.682390+0000","last_clean":"2024-03-23T15:47:28.682390+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:47:28.682390+0000","last_undegraded":"2024-03-23T15:47:28.682390+0000","last_fullsized":"2024-03-23T15:47:28.682390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.081361+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:47:38.081361+0000","last_peered":"2024-03-23T15:47:38.081361+0000","last_clean":"2024-03-23T15:47:38.081361+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:47:38.081361+0000","last_undegraded":"2024-03-23T15:47:38.081361+0000","last_fullsized":"2024-03-23T15:47:38.081361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747808+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:47:28.747808+0000","last_peered":"2024-03-23T15:47:28.747808+0000","last_clean":"2024-03-23T15:47:28.747808+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:47:28.747808+0000","last_undegraded":"2024-03-23T15:47:28.747808+0000","last_fullsized":"2024-03-23T15:47:28.747808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.082970+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:47:38.082970+0000","last_peered":"2024-03-23T15:47:38.082970+0000","last_clean":"2024-03-23T15:47:38.082970+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:47:38.082970+0000","last_undegraded":"2024-03-23T15:47:38.082970+0000","last_fullsized":"2024-03-23T15:47:38.082970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680240+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:47:28.680240+0000","last_peered":"2024-03-23T15:47:28.680240+0000","last_clean":"2024-03-23T15:47:28.680240+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:47:28.680240+0000","last_undegraded":"2024-03-23T15:47:28.680240+0000","last_fullsized":"2024-03-23T15:47:28.680240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.911567+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:47:39.911567+0000","last_peered":"2024-03-23T15:47:39.911567+0000","last_clean":"2024-03-23T15:47:39.911567+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:47:39.911567+0000","last_undegraded":"2024-03-23T15:47:39.911567+0000","last_fullsized":"2024-03-23T15:47:39.911567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.680148+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:47:28.680148+0000","last_peered":"2024-03-23T15:47:28.680148+0000","last_clean":"2024-03-23T15:47:28.680148+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:47:28.680148+0000","last_undegraded":"2024-03-23T15:47:28.680148+0000","last_fullsized":"2024-03-23T15:47:28.680148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.602195+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:47:33.602195+0000","last_peered":"2024-03-23T15:47:33.602195+0000","last_clean":"2024-03-23T15:47:33.602195+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:47:33.602195+0000","last_undegraded":"2024-03-23T15:47:33.602195+0000","last_fullsized":"2024-03-23T15:47:33.602195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416764+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:47:28.416764+0000","last_peered":"2024-03-23T15:47:28.416764+0000","last_clean":"2024-03-23T15:47:28.416764+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:47:28.416764+0000","last_undegraded":"2024-03-23T15:47:28.416764+0000","last_fullsized":"2024-03-23T15:47:28.416764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.910489+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:47:39.910489+0000","last_peered":"2024-03-23T15:47:39.910489+0000","last_clean":"2024-03-23T15:47:39.910489+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:47:39.910489+0000","last_undegraded":"2024-03-23T15:47:39.910489+0000","last_fullsized":"2024-03-23T15:47:39.910489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.679874+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:47:28.679874+0000","last_peered":"2024-03-23T15:47:28.679874+0000","last_clean":"2024-03-23T15:47:28.679874+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:47:28.679874+0000","last_undegraded":"2024-03-23T15:47:28.679874+0000","last_fullsized":"2024-03-23T15:47:28.679874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:38.077900+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:47:38.077900+0000","last_peered":"2024-03-23T15:47:38.077900+0000","last_clean":"2024-03-23T15:47:38.077900+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:47:38.077900+0000","last_undegraded":"2024-03-23T15:47:38.077900+0000","last_fullsized":"2024-03-23T15:47:38.077900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.682112+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:47:28.682112+0000","last_peered":"2024-03-23T15:47:28.682112+0000","last_clean":"2024-03-23T15:47:28.682112+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:47:28.682112+0000","last_undegraded":"2024-03-23T15:47:28.682112+0000","last_fullsized":"2024-03-23T15:47:28.682112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:33.493555+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:47:33.493555+0000","last_peered":"2024-03-23T15:47:33.493555+0000","last_clean":"2024-03-23T15:47:33.493555+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:47:33.493555+0000","last_undegraded":"2024-03-23T15:47:33.493555+0000","last_fullsized":"2024-03-23T15:47:33.493555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416913+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:47:28.416913+0000","last_peered":"2024-03-23T15:47:28.416913+0000","last_clean":"2024-03-23T15:47:28.416913+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:47:28.416913+0000","last_undegraded":"2024-03-23T15:47:28.416913+0000","last_fullsized":"2024-03-23T15:47:28.416913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.416753+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:47:28.416753+0000","last_peered":"2024-03-23T15:47:28.416753+0000","last_clean":"2024-03-23T15:47:28.416753+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:47:28.416753+0000","last_undegraded":"2024-03-23T15:47:28.416753+0000","last_fullsized":"2024-03-23T15:47:28.416753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.908492+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:47:39.908492+0000","last_peered":"2024-03-23T15:47:39.908492+0000","last_clean":"2024-03-23T15:47:39.908492+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:47:39.908492+0000","last_undegraded":"2024-03-23T15:47:39.908492+0000","last_fullsized":"2024-03-23T15:47:39.908492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.465591+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:47:28.465591+0000","last_peered":"2024-03-23T15:47:28.465591+0000","last_clean":"2024-03-23T15:47:28.465591+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:47:28.465591+0000","last_undegraded":"2024-03-23T15:47:28.465591+0000","last_fullsized":"2024-03-23T15:47:28.465591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.907672+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:47:39.907672+0000","last_peered":"2024-03-23T15:47:39.907672+0000","last_clean":"2024-03-23T15:47:39.907672+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:47:39.907672+0000","last_undegraded":"2024-03-23T15:47:39.907672+0000","last_fullsized":"2024-03-23T15:47:39.907672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411323+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:47:28.411323+0000","last_peered":"2024-03-23T15:47:28.411323+0000","last_clean":"2024-03-23T15:47:28.411323+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:47:28.411323+0000","last_undegraded":"2024-03-23T15:47:28.411323+0000","last_fullsized":"2024-03-23T15:47:28.411323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.747051+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:47:28.747051+0000","last_peered":"2024-03-23T15:47:28.747051+0000","last_clean":"2024-03-23T15:47:28.747051+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:47:28.747051+0000","last_undegraded":"2024-03-23T15:47:28.747051+0000","last_fullsized":"2024-03-23T15:47:28.747051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:28.411212+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:47:28.411212+0000","last_peered":"2024-03-23T15:47:28.411212+0000","last_clean":"2024-03-23T15:47:28.411212+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:47:28.411212+0000","last_undegraded":"2024-03-23T15:47:28.411212+0000","last_fullsized":"2024-03-23T15:47:28.411212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-23T15:47:39.910233+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:47:39.910233+0000","last_peered":"2024-03-23T15:47:39.910233+0000","last_clean":"2024-03-23T15:47:39.910233+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:47:39.910233+0000","last_undegraded":"2024-03-23T15:47:39.910233+0000","last_fullsized":"2024-03-23T15:47:39.910233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":77045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":78371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":667185,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":79441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":78713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":667773,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":78533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T15:47:44.668 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T15:47:44.668 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-23T15:47:44.668 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-23T15:47:44.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T15:47:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.802+0000 7f8a123e9640 1 Processor -- start 2024-03-23T15:47:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.803+0000 7f8a123e9640 1 -- start start 2024-03-23T15:47:44.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.803+0000 7f8a123e9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8a0c06cf70 0x7f8a0c0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.803+0000 7f8a123e9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8a0c15df80 0x7f8a0c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.803+0000 7f8a123e9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0c070640 con 0x7f8a0c15dba0 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.803+0000 7f8a123e9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8a0c0c4c00 con 0x7f8a0c06cf70 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.803+0000 7f8a123e9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8a0c06c240 con 0x7f8a0c15df80 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.804+0000 7f8a0b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8a0c06cf70 0x7f8a0c0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.804+0000 7f8a10986640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8a0c15df80 0x7f8a0c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.804+0000 7f8a0bfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8a0c15dba0 0x7f8a0c06ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44720/0 (socket says 172.21.15.129:44720) 2024-03-23T15:47:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.804+0000 7f8a0bfff640 1 -- 172.21.15.129:0/2289492306 learned_addr learned my addr 172.21.15.129:0/2289492306 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.807+0000 7f8a0b7fe640 1 -- 172.21.15.129:0/2289492306 >> v2:172.21.15.157:3300/0 conn(0x7f8a0c15df80 msgr2=0x7f8a0c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.807+0000 7f8a0b7fe640 1 --2- 172.21.15.129:0/2289492306 >> v2:172.21.15.157:3300/0 conn(0x7f8a0c15df80 0x7f8a0c1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.807+0000 7f8a0b7fe640 1 -- 172.21.15.129:0/2289492306 >> v1:172.21.15.129:6789/0 conn(0x7f8a0c15dba0 legacy=0x7f8a0c06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.807+0000 7f8a0b7fe640 1 -- 172.21.15.129:0/2289492306 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c0cbb20 con 0x7f8a0c06cf70 2024-03-23T15:47:44.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.811+0000 7f8a0b7fe640 1 --2- 172.21.15.129:0/2289492306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c0cca30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009790 tx=0x7f8a0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb8710caf2bba77 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:44.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.816+0000 7f8a0affd640 1 -- 172.21.15.129:0/2289492306 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00030610 con 0x7f8a0c06cf70 2024-03-23T15:47:44.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.816+0000 7f8a0affd640 1 -- 172.21.15.129:0/2289492306 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a00030770 con 0x7f8a0c06cf70 2024-03-23T15:47:44.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.816+0000 7f8a0affd640 1 -- 172.21.15.129:0/2289492306 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00030b00 con 0x7f8a0c06cf70 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 -- 172.21.15.129:0/2289492306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 msgr2=0x7f8a0c0cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 --2- 172.21.15.129:0/2289492306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c0cca30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009790 tx=0x7f8a0002b960 comp rx=0 tx=0).stop 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 -- 172.21.15.129:0/2289492306 shutdown_connections 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 --2- 172.21.15.129:0/2289492306 >> v2:172.21.15.157:3300/0 conn(0x7f8a0c15df80 0x7f8a0c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 --2- 172.21.15.129:0/2289492306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c0cca30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 -- 172.21.15.129:0/2289492306 >> 172.21.15.129:0/2289492306 conn(0x7f8a0c071300 msgr2=0x7f8a0c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 -- 172.21.15.129:0/2289492306 shutdown_connections 2024-03-23T15:47:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.817+0000 7f8a123e9640 1 -- 172.21.15.129:0/2289492306 wait complete. 2024-03-23T15:47:44.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.819+0000 7f8a123e9640 1 Processor -- start 2024-03-23T15:47:44.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.819+0000 7f8a123e9640 1 -- start start 2024-03-23T15:47:44.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.820+0000 7f8a123e9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.820+0000 7f8a123e9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15dba0 0x7f8a0c0c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.820+0000 7f8a123e9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c15df80 0x7f8a0c14ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:44.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.820+0000 7f8a123e9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1f09d0 con 0x7f8a0c06cf70 2024-03-23T15:47:44.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.820+0000 7f8a123e9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c1f4600 con 0x7f8a0c15df80 2024-03-23T15:47:44.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.820+0000 7f8a123e9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c1f0680 con 0x7f8a0c15dba0 2024-03-23T15:47:44.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.821+0000 7f8a0bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.821+0000 7f8a0b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15dba0 0x7f8a0c0c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.821+0000 7f8a10986640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c15df80 0x7f8a0c14ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:44.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.821+0000 7f8a0bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60766/0 (socket says 172.21.15.129:60766) 2024-03-23T15:47:44.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.821+0000 7f8a0bfff640 1 -- 172.21.15.129:0/582748991 learned_addr learned my addr 172.21.15.129:0/582748991 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:44.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.825+0000 7f8a0bfff640 1 -- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15dba0 msgr2=0x7f8a0c0c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.825+0000 7f8a0bfff640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15dba0 0x7f8a0c0c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.825+0000 7f8a0bfff640 1 -- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c15df80 msgr2=0x7f8a0c14ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:44.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.825+0000 7f8a0bfff640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c15df80 0x7f8a0c14ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:44.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.825+0000 7f8a0bfff640 1 -- 172.21.15.129:0/582748991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a000093a0 con 0x7f8a0c06cf70 2024-03-23T15:47:44.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.827+0000 7f8a0bfff640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c157910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c400 tx=0x7f89fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.831+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc005ce0 con 0x7f8a0c06cf70 2024-03-23T15:47:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.831+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89fc005e40 con 0x7f8a0c06cf70 2024-03-23T15:47:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.831+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0072d0 con 0x7f8a0c06cf70 2024-03-23T15:47:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.831+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c155e70 con 0x7f8a0c06cf70 2024-03-23T15:47:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.831+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c14b320 con 0x7f8a0c06cf70 2024-03-23T15:47:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:44.832+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0c156390 con 0x7f8a0c06cf70 2024-03-23T15:47:45.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.107+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f89fc0074f0 con 0x7f8a0c06cf70 2024-03-23T15:47:45.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.107+0000 7f8a08ff9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f89f806b860 0x7f89f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:45.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.108+0000 7f8a0b7fe640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f89f806b860 0x7f89f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:45.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.108+0000 7f8a0b7fe640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f89f806b860 0x7f89f806dd20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8a00036000 tx=0x7f8a00008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:45.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.121+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f89fc0583f0 con 0x7f8a0c06cf70 2024-03-23T15:47:45.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.121+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f89fc054690 con 0x7f8a0c06cf70 2024-03-23T15:47:45.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.359+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8a0c075740 con 0x7f8a0c06cf70 2024-03-23T15:47:45.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.365+0000 7f8a08ff9640 1 -- 172.21.15.129:0/582748991 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f89fc017080 con 0x7f8a0c06cf70 2024-03-23T15:47:45.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:45.369 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f89f806b860 msgr2=0x7f89f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f89f806b860 0x7f89f806dd20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8a00036000 tx=0x7f8a00008030 comp rx=0 tx=0).stop 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 msgr2=0x7f8a0c157910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c157910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c400 tx=0x7f89fc00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 shutdown_connections 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f89f806b860 0x7f89f806dd20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c15df80 0x7f8a0c14ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15dba0 0x7f8a0c0c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 --2- 172.21.15.129:0/582748991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a0c06cf70 0x7f8a0c157910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 >> 172.21.15.129:0/582748991 conn(0x7f8a0c071300 msgr2=0x7f8a0c15c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:45.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 shutdown_connections 2024-03-23T15:47:45.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.368+0000 7f8a123e9640 1 -- 172.21.15.129:0/582748991 wait complete. 2024-03-23T15:47:45.382 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-23T15:47:45.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:45.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.489+0000 7fcbdc49d640 1 Processor -- start 2024-03-23T15:47:45.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.490+0000 7fcbdc49d640 1 -- start start 2024-03-23T15:47:45.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.491+0000 7fcbdc49d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcbd415a2e0 0x7fcbd406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:45.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.491+0000 7fcbdc49d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd40cbcb0 con 0x7fcbd41544b0 2024-03-23T15:47:45.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.491+0000 7fcbdc49d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd415b280 con 0x7fcbd406c0b0 2024-03-23T15:47:45.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.491+0000 7fcbdc49d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd40c5be0 con 0x7fcbd415a2e0 2024-03-23T15:47:45.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.492+0000 7fcbda239640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fcbd41544b0 0x7fcbd40caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44722/0 (socket says 172.21.15.129:44722) 2024-03-23T15:47:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.492+0000 7fcbda239640 1 -- 172.21.15.129:0/126234740 learned_addr learned my addr 172.21.15.129:0/126234740 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.492+0000 7fcbd9a38640 1 --2- 172.21.15.129:0/126234740 >> v2:172.21.15.157:3300/0 conn(0x7fcbd415a2e0 0x7fcbd406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:45.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.495+0000 7fcbd9237640 1 -- 172.21.15.129:0/126234740 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337135425 0 0) 0x7fcbd40cbcb0 con 0x7fcbd41544b0 2024-03-23T15:47:45.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.495+0000 7fcbd9237640 1 -- 172.21.15.129:0/126234740 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbc0003520 con 0x7fcbd41544b0 2024-03-23T15:47:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.496+0000 7fcbd9a38640 1 -- 172.21.15.129:0/126234740 >> v1:172.21.15.157:6789/0 conn(0x7fcbd406c0b0 legacy=0x7fcbd40c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.496+0000 7fcbd9a38640 1 -- 172.21.15.129:0/126234740 >> v1:172.21.15.129:6789/0 conn(0x7fcbd41544b0 legacy=0x7fcbd40caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.496+0000 7fcbd9a38640 1 -- 172.21.15.129:0/126234740 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd415a9e0 con 0x7fcbd415a2e0 2024-03-23T15:47:45.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.501+0000 7fcbd9a38640 1 --2- 172.21.15.129:0/126234740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd415a2e0 0x7fcbd406bb70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40097a0 tx=0x7fcbc402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5511c3a8ae7734c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:45.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.507+0000 7fcbd9237640 1 -- 172.21.15.129:0/126234740 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4030610 con 0x7fcbd415a2e0 2024-03-23T15:47:45.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.507+0000 7fcbd9237640 1 -- 172.21.15.129:0/126234740 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc4030770 con 0x7fcbd415a2e0 2024-03-23T15:47:45.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.507+0000 7fcbd9237640 1 -- 172.21.15.129:0/126234740 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4030b00 con 0x7fcbd415a2e0 2024-03-23T15:47:45.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 -- 172.21.15.129:0/126234740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd415a2e0 msgr2=0x7fcbd406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:45.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/126234740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd415a2e0 0x7fcbd406bb70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40097a0 tx=0x7fcbc402b960 comp rx=0 tx=0).stop 2024-03-23T15:47:45.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 -- 172.21.15.129:0/126234740 shutdown_connections 2024-03-23T15:47:45.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/126234740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd415a2e0 0x7fcbd406bb70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 -- 172.21.15.129:0/126234740 >> 172.21.15.129:0/126234740 conn(0x7fcbd4061b50 msgr2=0x7fcbd4061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:45.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 -- 172.21.15.129:0/126234740 shutdown_connections 2024-03-23T15:47:45.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.508+0000 7fcbdc49d640 1 -- 172.21.15.129:0/126234740 wait complete. 2024-03-23T15:47:45.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.509+0000 7fcbdc49d640 1 Processor -- start 2024-03-23T15:47:45.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.509+0000 7fcbdc49d640 1 -- start start 2024-03-23T15:47:45.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.510+0000 7fcbdc49d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd406c0b0 0x7fcbd41584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.510+0000 7fcbdc49d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd41544b0 0x7fcbd41f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.510+0000 7fcbdc49d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbd415a2e0 0x7fcbd4150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.510+0000 7fcbdc49d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd414d270 con 0x7fcbd406c0b0 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.510+0000 7fcbdc49d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd41effe0 con 0x7fcbd41544b0 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.510+0000 7fcbdc49d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd41f1100 con 0x7fcbd415a2e0 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.511+0000 7fcbdaa3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbd415a2e0 0x7fcbd4150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.511+0000 7fcbd9a38640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd41544b0 0x7fcbd41f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.511+0000 7fcbdaa3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbd415a2e0 0x7fcbd4150a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48922/0 (socket says 172.21.15.129:48922) 2024-03-23T15:47:45.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.511+0000 7fcbdaa3a640 1 -- 172.21.15.129:0/507884464 learned_addr learned my addr 172.21.15.129:0/507884464 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:45.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.512+0000 7fcbda239640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd406c0b0 0x7fcbd41584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:45.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.516+0000 7fcbd9a38640 1 -- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbd415a2e0 msgr2=0x7fcbd4150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:45.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.516+0000 7fcbd9a38640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbd415a2e0 0x7fcbd4150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.516+0000 7fcbd9a38640 1 -- 172.21.15.129:0/507884464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd406c0b0 msgr2=0x7fcbd41584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:45.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.516+0000 7fcbd9a38640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd406c0b0 0x7fcbd41584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:45.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.516+0000 7fcbd9a38640 1 -- 172.21.15.129:0/507884464 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc4009400 con 0x7fcbd41544b0 2024-03-23T15:47:45.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.516+0000 7fcbda239640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd406c0b0 0x7fcbd41584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:47:45.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.519+0000 7fcbd9a38640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd41544b0 0x7fcbd41f0470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4033000 tx=0x7fcbc40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:45.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.524+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc403e070 con 0x7fcbd41544b0 2024-03-23T15:47:45.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.524+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc4030e30 con 0x7fcbd41544b0 2024-03-23T15:47:45.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.524+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd41efc90 con 0x7fcbd41544b0 2024-03-23T15:47:45.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.524+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4039c50 con 0x7fcbd41544b0 2024-03-23T15:47:45.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.524+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd41f39d0 con 0x7fcbd41544b0 2024-03-23T15:47:45.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.525+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd41f0150 con 0x7fcbd41544b0 2024-03-23T15:47:45.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.792+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcbc40423f0 con 0x7fcbd41544b0 2024-03-23T15:47:45.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.793+0000 7fcbcaffd640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcbb806b840 0x7fcbb806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:45.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.793+0000 7fcbda239640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcbb806b840 0x7fcbb806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:45.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.793+0000 7fcbda239640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcbb806b840 0x7fcbb806dd00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0005880 tx=0x7fcbd0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.797+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcbc407ae90 con 0x7fcbd41544b0 2024-03-23T15:47:45.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.815+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcbc4077130 con 0x7fcbd41544b0 2024-03-23T15:47:45.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:45.997+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbd4075740 con 0x7fcbd41544b0 2024-03-23T15:47:46.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.035+0000 7fcbcaffd640 1 -- 172.21.15.129:0/507884464 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fcbc4034090 con 0x7fcbd41544b0 2024-03-23T15:47:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:46.039 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:46.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcbb806b840 msgr2=0x7fcbb806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcbb806b840 0x7fcbb806dd00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0005880 tx=0x7fcbd0009040 comp rx=0 tx=0).stop 2024-03-23T15:47:46.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd41544b0 msgr2=0x7fcbd41f0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd41544b0 0x7fcbd41f0470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4033000 tx=0x7fcbc40046c0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 shutdown_connections 2024-03-23T15:47:46.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbd415a2e0 0x7fcbd4150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcbb806b840 0x7fcbb806dd00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbd41544b0 0x7fcbd41f0470 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 --2- 172.21.15.129:0/507884464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbd406c0b0 0x7fcbd41584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 >> 172.21.15.129:0/507884464 conn(0x7fcbd4061b50 msgr2=0x7fcbd40b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:46.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 shutdown_connections 2024-03-23T15:47:46.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.037+0000 7fcbdc49d640 1 -- 172.21.15.129:0/507884464 wait complete. 2024-03-23T15:47:46.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T15:47:11.061293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T15:47:24.638536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T15:47:25.617795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T15:47:46.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T15:47:46.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.158+0000 7f67a0c64640 1 Processor -- start 2024-03-23T15:47:46.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.158+0000 7f67a0c64640 1 -- start start 2024-03-23T15:47:46.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.159+0000 7f67a0c64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c15b940 0x7f679c06c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.159+0000 7f67a0c64640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f679c06d570 0x7f679c0cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.159+0000 7f67a0c64640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679c154c90 con 0x7f679c15d4f0 2024-03-23T15:47:46.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.159+0000 7f67a0c64640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f679c06cb20 con 0x7f679c06d570 2024-03-23T15:47:46.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.159+0000 7f67a0c64640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f679c06bb90 con 0x7f679c15b940 2024-03-23T15:47:46.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.160+0000 7f679a59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f679c15d4f0 0x7f679c15a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44724/0 (socket says 172.21.15.129:44724) 2024-03-23T15:47:46.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.160+0000 7f679a59c640 1 -- 172.21.15.129:0/50381605 learned_addr learned my addr 172.21.15.129:0/50381605 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:46.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.160+0000 7f6793fff640 1 --2- 172.21.15.129:0/50381605 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c15b940 0x7f679c06c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.160+0000 7f679ad9d640 1 --2- 172.21.15.129:0/50381605 >> v2:172.21.15.157:3300/0 conn(0x7f679c06d570 0x7f679c0cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.162+0000 7f6799d9b640 1 -- 172.21.15.129:0/50381605 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009892943 0 0) 0x7f679c154c90 con 0x7f679c15d4f0 2024-03-23T15:47:46.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.162+0000 7f6799d9b640 1 -- 172.21.15.129:0/50381605 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6788003520 con 0x7f679c15d4f0 2024-03-23T15:47:46.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.163+0000 7f6793fff640 1 -- 172.21.15.129:0/50381605 >> v2:172.21.15.157:3300/0 conn(0x7f679c06d570 msgr2=0x7f679c0cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:46.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.163+0000 7f6793fff640 1 --2- 172.21.15.129:0/50381605 >> v2:172.21.15.157:3300/0 conn(0x7f679c06d570 0x7f679c0cc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.163+0000 7f6793fff640 1 -- 172.21.15.129:0/50381605 >> v1:172.21.15.129:6789/0 conn(0x7f679c15d4f0 legacy=0x7f679c15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.163+0000 7f6793fff640 1 -- 172.21.15.129:0/50381605 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679c06c150 con 0x7f679c15b940 2024-03-23T15:47:46.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.166+0000 7f6793fff640 1 --2- 172.21.15.129:0/50381605 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c15b940 0x7f679c06c5e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f67840097a0 tx=0x7f678402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514f03db6fd1c9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:46.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.171+0000 7f6799d9b640 1 -- 172.21.15.129:0/50381605 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784030610 con 0x7f679c15b940 2024-03-23T15:47:46.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.171+0000 7f6799d9b640 1 -- 172.21.15.129:0/50381605 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6784030770 con 0x7f679c15b940 2024-03-23T15:47:46.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.171+0000 7f6799d9b640 1 -- 172.21.15.129:0/50381605 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784030b00 con 0x7f679c15b940 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 -- 172.21.15.129:0/50381605 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c15b940 msgr2=0x7f679c06c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 --2- 172.21.15.129:0/50381605 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c15b940 0x7f679c06c5e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f67840097a0 tx=0x7f678402b960 comp rx=0 tx=0).stop 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 -- 172.21.15.129:0/50381605 shutdown_connections 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 --2- 172.21.15.129:0/50381605 >> v2:172.21.15.157:3300/0 conn(0x7f679c06d570 0x7f679c0cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 --2- 172.21.15.129:0/50381605 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c15b940 0x7f679c06c5e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 -- 172.21.15.129:0/50381605 >> 172.21.15.129:0/50381605 conn(0x7f679c0cfac0 msgr2=0x7f679c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 -- 172.21.15.129:0/50381605 shutdown_connections 2024-03-23T15:47:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.172+0000 7f67a0c64640 1 -- 172.21.15.129:0/50381605 wait complete. 2024-03-23T15:47:46.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.174+0000 7f67a0c64640 1 Processor -- start 2024-03-23T15:47:46.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.174+0000 7f67a0c64640 1 -- start start 2024-03-23T15:47:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.175+0000 7f67a0c64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c06d570 0x7f679c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.175+0000 7f67a0c64640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f679c15b940 0x7f679c155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.175+0000 7f67a0c64640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 0x7f679c155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.175+0000 7f67a0c64640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f679c156480 con 0x7f679c15d4f0 2024-03-23T15:47:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.175+0000 7f67a0c64640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f679c0c48c0 con 0x7f679c15b940 2024-03-23T15:47:46.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.175+0000 7f67a0c64640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f679c1f09f0 con 0x7f679c06d570 2024-03-23T15:47:46.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.176+0000 7f679ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 0x7f679c155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.176+0000 7f6793fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f679c15b940 0x7f679c155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.176+0000 7f679a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c06d570 0x7f679c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.176+0000 7f679ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 0x7f679c155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60790/0 (socket says 172.21.15.129:60790) 2024-03-23T15:47:46.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.176+0000 7f679ad9d640 1 -- 172.21.15.129:0/644789243 learned_addr learned my addr 172.21.15.129:0/644789243 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.179+0000 7f679ad9d640 1 -- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c06d570 msgr2=0x7f679c14d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.179+0000 7f679ad9d640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c06d570 0x7f679c14d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.179+0000 7f679ad9d640 1 -- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f679c15b940 msgr2=0x7f679c155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.179+0000 7f679ad9d640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f679c15b940 0x7f679c155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.179+0000 7f679ad9d640 1 -- 172.21.15.129:0/644789243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6784009400 con 0x7f679c15d4f0 2024-03-23T15:47:46.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.182+0000 7f679ad9d640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 0x7f679c155d70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f679400efc0 tx=0x7f679400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.186+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794014480 con 0x7f679c15d4f0 2024-03-23T15:47:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.186+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f679400f040 con 0x7f679c15d4f0 2024-03-23T15:47:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.186+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794014820 con 0x7f679c15d4f0 2024-03-23T15:47:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.186+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679c1f0cd0 con 0x7f679c15d4f0 2024-03-23T15:47:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.186+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f679c0c4df0 con 0x7f679c15d4f0 2024-03-23T15:47:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.187+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f679c0c4fe0 con 0x7f679c15d4f0 2024-03-23T15:47:46.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.450+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6794014980 con 0x7f679c15d4f0 2024-03-23T15:47:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.451+0000 7f6792ffd640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f677c06b860 0x7f677c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.451+0000 7f679a59c640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f677c06b860 0x7f677c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.451+0000 7f679a59c640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f677c06b860 0x7f677c06dd20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f678c005880 tx=0x7f678c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.464+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6794059d00 con 0x7f679c15d4f0 2024-03-23T15:47:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.464+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6794055fa0 con 0x7f679c15d4f0 2024-03-23T15:47:46.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.664+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f679c075740 con 0x7f679c15d4f0 2024-03-23T15:47:46.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.693+0000 7f6792ffd640 1 -- 172.21.15.129:0/644789243 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f6794018070 con 0x7f679c15d4f0 2024-03-23T15:47:46.696 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-03-23T15:47:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f677c06b860 msgr2=0x7f677c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f677c06b860 0x7f677c06dd20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f678c005880 tx=0x7f678c009040 comp rx=0 tx=0).stop 2024-03-23T15:47:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 msgr2=0x7f679c155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 0x7f679c155d70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f679400efc0 tx=0x7f679400c490 comp rx=0 tx=0).stop 2024-03-23T15:47:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 shutdown_connections 2024-03-23T15:47:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f679c15d4f0 0x7f679c155d70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f677c06b860 0x7f677c06dd20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f679c15b940 0x7f679c155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 --2- 172.21.15.129:0/644789243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f679c06d570 0x7f679c14d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 >> 172.21.15.129:0/644789243 conn(0x7f679c0cfac0 msgr2=0x7f679c0c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:46.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 shutdown_connections 2024-03-23T15:47:46.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.695+0000 7f67a0c64640 1 -- 172.21.15.129:0/644789243 wait complete. 2024-03-23T15:47:46.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T15:47:46.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.813+0000 7f0fdcc4e640 1 Processor -- start 2024-03-23T15:47:46.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.813+0000 7f0fdcc4e640 1 -- start start 2024-03-23T15:47:46.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.814+0000 7f0fdcc4e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0fd815d4f0 0x7f0fd806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.814+0000 7f0fdcc4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 0x7f0fd815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.814+0000 7f0fdcc4e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd8154c90 con 0x7f0fd806d570 2024-03-23T15:47:46.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.814+0000 7f0fdcc4e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd806cb20 con 0x7f0fd815d4f0 2024-03-23T15:47:46.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.814+0000 7f0fdcc4e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd806bb90 con 0x7f0fd815b940 2024-03-23T15:47:46.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.815+0000 7f0fd659c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0fd815d4f0 0x7f0fd806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.815+0000 7f0fd5d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 0x7f0fd815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.815+0000 7f0fd5d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 0x7f0fd815a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48936/0 (socket says 172.21.15.129:48936) 2024-03-23T15:47:46.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.815+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/2373669535 learned_addr learned my addr 172.21.15.129:0/2373669535 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:46.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.818+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/2373669535 >> v2:172.21.15.129:3300/0 conn(0x7f0fd815d4f0 msgr2=0x7f0fd806c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:46.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.818+0000 7f0fd5d9b640 1 --2- 172.21.15.129:0/2373669535 >> v2:172.21.15.129:3300/0 conn(0x7f0fd815d4f0 0x7f0fd806c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.818+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/2373669535 >> v1:172.21.15.129:6789/0 conn(0x7f0fd806d570 legacy=0x7f0fd80cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.818+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/2373669535 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd806c150 con 0x7f0fd815b940 2024-03-23T15:47:46.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.821+0000 7f0fd5d9b640 1 --2- 172.21.15.129:0/2373669535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 0x7f0fd815a4d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009870 tx=0x7f0fcc02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34c0983dcd947555 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:46.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.826+0000 7f0fd559a640 1 -- 172.21.15.129:0/2373669535 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc02f840 con 0x7f0fd815b940 2024-03-23T15:47:46.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.826+0000 7f0fd559a640 1 -- 172.21.15.129:0/2373669535 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc02f9a0 con 0x7f0fd815b940 2024-03-23T15:47:46.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.826+0000 7f0fd559a640 1 -- 172.21.15.129:0/2373669535 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc02fd30 con 0x7f0fd815b940 2024-03-23T15:47:46.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/2373669535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 msgr2=0x7f0fd815a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/2373669535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 0x7f0fd815a4d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009870 tx=0x7f0fcc02ba20 comp rx=0 tx=0).stop 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/2373669535 shutdown_connections 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/2373669535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815b940 0x7f0fd815a4d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/2373669535 >> v2:172.21.15.129:3300/0 conn(0x7f0fd815d4f0 0x7f0fd806c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/2373669535 >> 172.21.15.129:0/2373669535 conn(0x7f0fd80cfac0 msgr2=0x7f0fd80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/2373669535 shutdown_connections 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.827+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/2373669535 wait complete. 2024-03-23T15:47:46.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.829+0000 7f0fdcc4e640 1 Processor -- start 2024-03-23T15:47:46.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.829+0000 7f0fdcc4e640 1 -- start start 2024-03-23T15:47:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.830+0000 7f0fdcc4e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fd806d570 0x7f0fd814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.830+0000 7f0fdcc4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fd815b940 0x7f0fd814fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.830+0000 7f0fdcc4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815d4f0 0x7f0fd8150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.830+0000 7f0fdcc4e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd814e010 con 0x7f0fd815b940 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.830+0000 7f0fdcc4e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd81f2e60 con 0x7f0fd806d570 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.830+0000 7f0fdcc4e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd80c8a40 con 0x7f0fd815d4f0 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.831+0000 7f0fd5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fd815b940 0x7f0fd814fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.831+0000 7f0fd6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815d4f0 0x7f0fd8150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.831+0000 7f0fd659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fd806d570 0x7f0fd814dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.831+0000 7f0fd6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815d4f0 0x7f0fd8150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48952/0 (socket says 172.21.15.129:48952) 2024-03-23T15:47:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.831+0000 7f0fd6d9d640 1 -- 172.21.15.129:0/3101245284 learned_addr learned my addr 172.21.15.129:0/3101245284 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:46.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.835+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815d4f0 msgr2=0x7f0fd8150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.835+0000 7f0fd5d9b640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815d4f0 0x7f0fd8150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.835+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fd806d570 msgr2=0x7f0fd814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.835+0000 7f0fd5d9b640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fd806d570 0x7f0fd814dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.835+0000 7f0fd5d9b640 1 -- 172.21.15.129:0/3101245284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc0094d0 con 0x7f0fd815b940 2024-03-23T15:47:46.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.837+0000 7f0fd5d9b640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fd815b940 0x7f0fd814fc00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc033000 tx=0x7f0fcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.840+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc037bc0 con 0x7f0fd815b940 2024-03-23T15:47:46.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.840+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc037d20 con 0x7f0fd815b940 2024-03-23T15:47:46.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.840+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd814f770 con 0x7f0fd815b940 2024-03-23T15:47:46.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.840+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc041640 con 0x7f0fd815b940 2024-03-23T15:47:46.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.841+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd81514b0 con 0x7f0fd815b940 2024-03-23T15:47:46.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:46.842+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd814f900 con 0x7f0fd815b940 2024-03-23T15:47:47.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.097+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0fcc0417a0 con 0x7f0fd815b940 2024-03-23T15:47:47.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.097+0000 7f0fc6ffd640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb806b810 0x7f0fb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:47.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.098+0000 7f0fd659c640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb806b810 0x7f0fb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:47.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.098+0000 7f0fd659c640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb806b810 0x7f0fb806dcd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009aa0 tx=0x7f0fc8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.111+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0fcc03c070 con 0x7f0fd815b940 2024-03-23T15:47:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.111+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0fcc0783e0 con 0x7f0fd815b940 2024-03-23T15:47:47.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.290+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0fd8075740 con 0x7f0fd815b940 2024-03-23T15:47:47.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.297+0000 7f0fc6ffd640 1 -- 172.21.15.129:0/3101245284 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f0fcc07baa0 con 0x7f0fd815b940 2024-03-23T15:47:47.300 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-03-23T15:47:47.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb806b810 msgr2=0x7f0fb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb806b810 0x7f0fb806dcd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009aa0 tx=0x7f0fc8009190 comp rx=0 tx=0).stop 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fd815b940 msgr2=0x7f0fd814fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fd815b940 0x7f0fd814fc00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc033000 tx=0x7f0fcc004730 comp rx=0 tx=0).stop 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 shutdown_connections 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fd815d4f0 0x7f0fd8150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fd815b940 0x7f0fd814fc00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb806b810 0x7f0fb806dcd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 --2- 172.21.15.129:0/3101245284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fd806d570 0x7f0fd814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 >> 172.21.15.129:0/3101245284 conn(0x7f0fd80cfac0 msgr2=0x7f0fd80c7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 shutdown_connections 2024-03-23T15:47:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.300+0000 7f0fdcc4e640 1 -- 172.21.15.129:0/3101245284 wait complete. 2024-03-23T15:47:47.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T15:47:47.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.421+0000 7f9d79e83640 1 Processor -- start 2024-03-23T15:47:47.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.421+0000 7f9d79e83640 1 -- start start 2024-03-23T15:47:47.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.421+0000 7f9d79e83640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9d7415b940 0x7f9d7406c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:47.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.422+0000 7f9d79e83640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d74154c90 con 0x7f9d7406d570 2024-03-23T15:47:47.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.422+0000 7f9d79e83640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7406bb90 con 0x7f9d7415d4f0 2024-03-23T15:47:47.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.422+0000 7f9d79e83640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9d7406cb20 con 0x7f9d7415b940 2024-03-23T15:47:47.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.422+0000 7f9d72ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9d7415b940 0x7f9d7406c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:47.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.422+0000 7f9d737fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9d7415d4f0 0x7f9d7415a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:39808/0 (socket says 172.21.15.129:39808) 2024-03-23T15:47:47.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.422+0000 7f9d737fe640 1 -- 172.21.15.129:0/4200139685 learned_addr learned my addr 172.21.15.129:0/4200139685 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.426+0000 7f9d72ffd640 1 -- 172.21.15.129:0/4200139685 >> v1:172.21.15.157:6789/0 conn(0x7f9d7415d4f0 legacy=0x7f9d7415a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.426+0000 7f9d72ffd640 1 -- 172.21.15.129:0/4200139685 >> v1:172.21.15.129:6789/0 conn(0x7f9d7406d570 legacy=0x7f9d740cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.426+0000 7f9d72ffd640 1 -- 172.21.15.129:0/4200139685 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7406c150 con 0x7f9d7415b940 2024-03-23T15:47:47.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.431+0000 7f9d72ffd640 1 --2- 172.21.15.129:0/4200139685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d7406c5e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d680097a0 tx=0x7f9d6802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bc6aa8610e8538 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:47.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.434+0000 7f9d727fc640 1 -- 172.21.15.129:0/4200139685 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6802f810 con 0x7f9d7415b940 2024-03-23T15:47:47.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.434+0000 7f9d727fc640 1 -- 172.21.15.129:0/4200139685 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6802f970 con 0x7f9d7415b940 2024-03-23T15:47:47.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.434+0000 7f9d727fc640 1 -- 172.21.15.129:0/4200139685 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6802fd00 con 0x7f9d7415b940 2024-03-23T15:47:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.435+0000 7f9d79e83640 1 -- 172.21.15.129:0/4200139685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 msgr2=0x7f9d7406c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.435+0000 7f9d79e83640 1 --2- 172.21.15.129:0/4200139685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d7406c5e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d680097a0 tx=0x7f9d6802b9f0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.435+0000 7f9d79e83640 1 -- 172.21.15.129:0/4200139685 shutdown_connections 2024-03-23T15:47:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.436+0000 7f9d79e83640 1 --2- 172.21.15.129:0/4200139685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d7406c5e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.436+0000 7f9d79e83640 1 -- 172.21.15.129:0/4200139685 >> 172.21.15.129:0/4200139685 conn(0x7f9d740cfac0 msgr2=0x7f9d740cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.436+0000 7f9d79e83640 1 -- 172.21.15.129:0/4200139685 shutdown_connections 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.436+0000 7f9d79e83640 1 -- 172.21.15.129:0/4200139685 wait complete. 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.437+0000 7f9d79e83640 1 Processor -- start 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.437+0000 7f9d79e83640 1 -- start start 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d79e83640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d7406d570 0x7f9d740aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d79e83640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d740b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d79e83640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d7415d4f0 0x7f9d740c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d79e83640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d74157a00 con 0x7f9d7415b940 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d79e83640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9d740c39a0 con 0x7f9d7415d4f0 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d79e83640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9d740c6770 con 0x7f9d7406d570 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d72ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d740b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.438+0000 7f9d73fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d7415d4f0 0x7f9d740c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.439+0000 7f9d737fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d7406d570 0x7f9d740aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.439+0000 7f9d72ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d740b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60820/0 (socket says 172.21.15.129:60820) 2024-03-23T15:47:47.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.439+0000 7f9d72ffd640 1 -- 172.21.15.129:0/2854997474 learned_addr learned my addr 172.21.15.129:0/2854997474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:47.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-03-23T15:47:47.081+0000 f5f1640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-23T15:47:47.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-03-23T15:47:47.082+0000 f5f1640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-23T15:47:47.445 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-03-23T15:47:47.082+0000 f5f1640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-23T15:47:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.442+0000 7f9d72ffd640 1 -- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d7406d570 msgr2=0x7f9d740aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.442+0000 7f9d72ffd640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d7406d570 0x7f9d740aff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.442+0000 7f9d72ffd640 1 -- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d7415d4f0 msgr2=0x7f9d740c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.442+0000 7f9d72ffd640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d7415d4f0 0x7f9d740c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.442+0000 7f9d72ffd640 1 -- 172.21.15.129:0/2854997474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d68009400 con 0x7f9d7415b940 2024-03-23T15:47:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.444+0000 7f9d72ffd640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d740b1100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d680098c0 tx=0x7f9d68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:47.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.448+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6803c040 con 0x7f9d7415b940 2024-03-23T15:47:47.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.448+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d680379d0 con 0x7f9d7415b940 2024-03-23T15:47:47.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.448+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d68037d60 con 0x7f9d7415b940 2024-03-23T15:47:47.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.448+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d74156890 con 0x7f9d7415b940 2024-03-23T15:47:47.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.448+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d74156b30 con 0x7f9d7415b940 2024-03-23T15:47:47.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.449+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d740c3b50 con 0x7f9d7415b940 2024-03-23T15:47:47.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.470+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9d6804b3b0 con 0x7f9d7415b940 2024-03-23T15:47:47.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.470+0000 7f9d5bfff640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d6006b810 0x7f9d6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:47.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.471+0000 7f9d737fe640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d6006b810 0x7f9d6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:47.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.471+0000 7f9d737fe640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d6006b810 0x7f9d6006dcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9d64005880 tx=0x7f9d64009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.474+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9d6807ade0 con 0x7f9d7415b940 2024-03-23T15:47:47.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.718+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9d68077080 con 0x7f9d7415b940 2024-03-23T15:47:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.901+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9d740c3b10 con 0x7f9d7415b940 2024-03-23T15:47:47.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.909+0000 7f9d5bfff640 1 -- 172.21.15.129:0/2854997474 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f9d68047020 con 0x7f9d7415b940 2024-03-23T15:47:47.912 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d6006b810 msgr2=0x7f9d6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d6006b810 0x7f9d6006dcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9d64005880 tx=0x7f9d64009040 comp rx=0 tx=0).stop 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 msgr2=0x7f9d740b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d740b1100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d680098c0 tx=0x7f9d68004730 comp rx=0 tx=0).stop 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 shutdown_connections 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d6006b810 0x7f9d6006dcd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d7415d4f0 0x7f9d740c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d7415b940 0x7f9d740b1100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 --2- 172.21.15.129:0/2854997474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d7406d570 0x7f9d740aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 >> 172.21.15.129:0/2854997474 conn(0x7f9d740cfac0 msgr2=0x7f9d7415e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 shutdown_connections 2024-03-23T15:47:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:47.912+0000 7f9d79e83640 1 -- 172.21.15.129:0/2854997474 wait complete. 2024-03-23T15:47:47.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.028+0000 7fe53f7c9640 1 Processor -- start 2024-03-23T15:47:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 -- start start 2024-03-23T15:47:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe5381544b0 0x7fe53815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe5380caa00 0x7fe53806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe53806c0b0 0x7fe5380c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe5380c5be0 con 0x7fe53806c0b0 2024-03-23T15:47:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe5380cbcb0 con 0x7fe5381544b0 2024-03-23T15:47:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.029+0000 7fe53f7c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe53815b280 con 0x7fe5380caa00 2024-03-23T15:47:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.030+0000 7fe53dd66640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe53806c0b0 0x7fe5380c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.030+0000 7fe53cd64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe5380caa00 0x7fe53806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.030+0000 7fe53d565640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe5381544b0 0x7fe53815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.030+0000 7fe53dd66640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe53806c0b0 0x7fe5380c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60834/0 (socket says 172.21.15.129:60834) 2024-03-23T15:47:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.030+0000 7fe53dd66640 1 -- 172.21.15.129:0/177454116 learned_addr learned my addr 172.21.15.129:0/177454116 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.033+0000 7fe53dd66640 1 -- 172.21.15.129:0/177454116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe5380caa00 msgr2=0x7fe53806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.033+0000 7fe53dd66640 1 --2- 172.21.15.129:0/177454116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe5380caa00 0x7fe53806bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.033+0000 7fe53dd66640 1 -- 172.21.15.129:0/177454116 >> v2:172.21.15.157:3300/0 conn(0x7fe5381544b0 msgr2=0x7fe53815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.033+0000 7fe53dd66640 1 --2- 172.21.15.129:0/177454116 >> v2:172.21.15.157:3300/0 conn(0x7fe5381544b0 0x7fe53815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.033+0000 7fe53dd66640 1 -- 172.21.15.129:0/177454116 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53815a9e0 con 0x7fe53806c0b0 2024-03-23T15:47:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.034+0000 7fe53dd66640 1 --2- 172.21.15.129:0/177454116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe53806c0b0 0x7fe5380c6000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe53806cd70 tx=0x7fe53002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99d3206380ce707 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe52ffff640 1 -- 172.21.15.129:0/177454116 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53002cb00 con 0x7fe53806c0b0 2024-03-23T15:47:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe52ffff640 1 -- 172.21.15.129:0/177454116 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53002cc60 con 0x7fe53806c0b0 2024-03-23T15:47:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe52ffff640 1 -- 172.21.15.129:0/177454116 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5300343f0 con 0x7fe53806c0b0 2024-03-23T15:47:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 -- 172.21.15.129:0/177454116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe53806c0b0 msgr2=0x7fe5380c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/177454116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe53806c0b0 0x7fe5380c6000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe53806cd70 tx=0x7fe53002bf60 comp rx=0 tx=0).stop 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 -- 172.21.15.129:0/177454116 shutdown_connections 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/177454116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe53806c0b0 0x7fe5380c6000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/177454116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe5380caa00 0x7fe53806bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/177454116 >> v2:172.21.15.157:3300/0 conn(0x7fe5381544b0 0x7fe53815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.039+0000 7fe53f7c9640 1 -- 172.21.15.129:0/177454116 >> 172.21.15.129:0/177454116 conn(0x7fe538061b50 msgr2=0x7fe538061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.040+0000 7fe53f7c9640 1 -- 172.21.15.129:0/177454116 shutdown_connections 2024-03-23T15:47:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.040+0000 7fe53f7c9640 1 -- 172.21.15.129:0/177454116 wait complete. 2024-03-23T15:47:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.041+0000 7fe53f7c9640 1 Processor -- start 2024-03-23T15:47:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.041+0000 7fe53f7c9640 1 -- start start 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53f7c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe53806c0b0 0x7fe538148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53f7c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 0x7fe5380c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53f7c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5381544b0 0x7fe5381f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53f7c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe5380c73e0 con 0x7fe5380caa00 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53f7c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe5381556a0 con 0x7fe5381544b0 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53f7c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe5380c7ec0 con 0x7fe53806c0b0 2024-03-23T15:47:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53cd64640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 0x7fe5380c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53d565640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe53806c0b0 0x7fe538148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53dd66640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5381544b0 0x7fe5381f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53cd64640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 0x7fe5380c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60848/0 (socket says 172.21.15.129:60848) 2024-03-23T15:47:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.043+0000 7fe53cd64640 1 -- 172.21.15.129:0/2625866849 learned_addr learned my addr 172.21.15.129:0/2625866849 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.047+0000 7fe53cd64640 1 -- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe53806c0b0 msgr2=0x7fe538148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.047+0000 7fe53cd64640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe53806c0b0 0x7fe538148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.047+0000 7fe53cd64640 1 -- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5381544b0 msgr2=0x7fe5381f3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.047+0000 7fe53cd64640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5381544b0 0x7fe5381f3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.047+0000 7fe53cd64640 1 -- 172.21.15.129:0/2625866849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe530009b30 con 0x7fe5380caa00 2024-03-23T15:47:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.050+0000 7fe53cd64640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 0x7fe5380c75e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe52800e890 tx=0x7fe52800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.052+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52800c940 con 0x7fe5380caa00 2024-03-23T15:47:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.052+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe538154d70 con 0x7fe5380caa00 2024-03-23T15:47:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.052+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe538155330 con 0x7fe5380caa00 2024-03-23T15:47:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.053+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe538075740 con 0x7fe5380caa00 2024-03-23T15:47:48.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.306+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe52800caa0 con 0x7fe5380caa00 2024-03-23T15:47:48.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.306+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52800ce10 con 0x7fe5380caa00 2024-03-23T15:47:48.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.307+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe528013470 con 0x7fe5380caa00 2024-03-23T15:47:48.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.307+0000 7fe52dffb640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe51c06b840 0x7fe51c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.308+0000 7fe53d565640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe51c06b840 0x7fe51c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.308+0000 7fe53d565640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe51c06b840 0x7fe51c06dd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe534009b40 tx=0x7fe534009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:48.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.320+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe5280592b0 con 0x7fe5380caa00 2024-03-23T15:47:48.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.320+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe528055550 con 0x7fe5380caa00 2024-03-23T15:47:48.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.539+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe538155850 con 0x7fe5380caa00 2024-03-23T15:47:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.581+0000 7fe52dffb640 1 -- 172.21.15.129:0/2625866849 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6920 (secure 0 0 0) 0x7fe528021090 con 0x7fe5380caa00 2024-03-23T15:47:48.585 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:48.585 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4234,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:47:40.883381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4250,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6837/3877606926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3877606926},{"type":"v1","addr":"172.21.15.157:6837","nonce":3877606926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4250,4253]},"id":1}]} 2024-03-23T15:47:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe51c06b840 msgr2=0x7fe51c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe51c06b840 0x7fe51c06dd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe534009b40 tx=0x7fe534009190 comp rx=0 tx=0).stop 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 msgr2=0x7fe5380c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 0x7fe5380c75e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe52800e890 tx=0x7fe52800ed50 comp rx=0 tx=0).stop 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 shutdown_connections 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe51c06b840 0x7fe51c06dd00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe5381544b0 0x7fe5381f3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5380caa00 0x7fe5380c75e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 --2- 172.21.15.129:0/2625866849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe53806c0b0 0x7fe538148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 >> 172.21.15.129:0/2625866849 conn(0x7fe538061b50 msgr2=0x7fe538061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 shutdown_connections 2024-03-23T15:47:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.584+0000 7fe53f7c9640 1 -- 172.21.15.129:0/2625866849 wait complete. 2024-03-23T15:47:48.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:48.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 Processor -- start 2024-03-23T15:47:48.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 -- start start 2024-03-23T15:47:48.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 0x7f727c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f727c06c0b0 0x7f727c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f727c0cbcb0 con 0x7f727c1544b0 2024-03-23T15:47:48.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f727c0c5be0 con 0x7f727c06c0b0 2024-03-23T15:47:48.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.698+0000 7f7282df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f727c15b280 con 0x7f727c15a2e0 2024-03-23T15:47:48.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.699+0000 7f727bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 0x7f727c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.699+0000 7f7281391640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f727c06c0b0 0x7f727c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.699+0000 7f727bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 0x7f727c06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48976/0 (socket says 172.21.15.129:48976) 2024-03-23T15:47:48.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.699+0000 7f727bfff640 1 -- 172.21.15.129:0/425164895 learned_addr learned my addr 172.21.15.129:0/425164895 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.702+0000 7f727bfff640 1 -- 172.21.15.129:0/425164895 >> v2:172.21.15.129:3300/0 conn(0x7f727c06c0b0 msgr2=0x7f727c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.702+0000 7f727bfff640 1 --2- 172.21.15.129:0/425164895 >> v2:172.21.15.129:3300/0 conn(0x7f727c06c0b0 0x7f727c0c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.702+0000 7f727bfff640 1 -- 172.21.15.129:0/425164895 >> v1:172.21.15.129:6789/0 conn(0x7f727c1544b0 legacy=0x7f727c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.702+0000 7f727bfff640 1 -- 172.21.15.129:0/425164895 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727c15a9e0 con 0x7f727c15a2e0 2024-03-23T15:47:48.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.704+0000 7f727bfff640 1 --2- 172.21.15.129:0/425164895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 0x7f727c06bb70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7268009870 tx=0x7f726802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b99b96c54523712b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:48.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f727b7fe640 1 -- 172.21.15.129:0/425164895 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726802f840 con 0x7f727c15a2e0 2024-03-23T15:47:48.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f727b7fe640 1 -- 172.21.15.129:0/425164895 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726802f9a0 con 0x7f727c15a2e0 2024-03-23T15:47:48.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f727b7fe640 1 -- 172.21.15.129:0/425164895 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726802fd30 con 0x7f727c15a2e0 2024-03-23T15:47:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 -- 172.21.15.129:0/425164895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 msgr2=0x7f727c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 --2- 172.21.15.129:0/425164895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 0x7f727c06bb70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7268009870 tx=0x7f726802ba20 comp rx=0 tx=0).stop 2024-03-23T15:47:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 -- 172.21.15.129:0/425164895 shutdown_connections 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 --2- 172.21.15.129:0/425164895 >> v2:172.21.15.129:3300/0 conn(0x7f727c06c0b0 0x7f727c0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 --2- 172.21.15.129:0/425164895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c15a2e0 0x7f727c06bb70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 -- 172.21.15.129:0/425164895 >> 172.21.15.129:0/425164895 conn(0x7f727c061b50 msgr2=0x7f727c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 -- 172.21.15.129:0/425164895 shutdown_connections 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.708+0000 7f7282df4640 1 -- 172.21.15.129:0/425164895 wait complete. 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 Processor -- start 2024-03-23T15:47:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 -- start start 2024-03-23T15:47:48.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 0x7f727c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f727c1544b0 0x7f727c0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f727c15a2e0 0x7f727c1f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f727c147e90 con 0x7f727c1544b0 2024-03-23T15:47:48.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f727c146080 con 0x7f727c15a2e0 2024-03-23T15:47:48.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.710+0000 7f7282df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f727c146360 con 0x7f727c06c0b0 2024-03-23T15:47:48.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.711+0000 7f7280b90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 0x7f727c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.711+0000 7f727bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f727c1544b0 0x7f727c0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.711+0000 7f7280b90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 0x7f727c0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:48978/0 (socket says 172.21.15.129:48978) 2024-03-23T15:47:48.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.711+0000 7f7280b90640 1 -- 172.21.15.129:0/441686944 learned_addr learned my addr 172.21.15.129:0/441686944 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:48.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.711+0000 7f7281391640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f727c15a2e0 0x7f727c1f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.716+0000 7f7280b90640 1 -- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f727c15a2e0 msgr2=0x7f727c1f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.716+0000 7f7280b90640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f727c15a2e0 0x7f727c1f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.716+0000 7f7280b90640 1 -- 172.21.15.129:0/441686944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f727c1544b0 msgr2=0x7f727c0b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:48.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.716+0000 7f7280b90640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f727c1544b0 0x7f727c0b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:48.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.716+0000 7f7280b90640 1 -- 172.21.15.129:0/441686944 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72680094d0 con 0x7f727c06c0b0 2024-03-23T15:47:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.719+0000 7f7280b90640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 0x7f727c0b3e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f727000c400 tx=0x7f727000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.721+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72700052d0 con 0x7f727c06c0b0 2024-03-23T15:47:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.721+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727c146640 con 0x7f727c06c0b0 2024-03-23T15:47:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.722+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f727c1f11a0 con 0x7f727c06c0b0 2024-03-23T15:47:48.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.723+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727c146b60 con 0x7f727c06c0b0 2024-03-23T15:47:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.990+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7270005430 con 0x7f727c06c0b0 2024-03-23T15:47:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.990+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270007240 con 0x7f727c06c0b0 2024-03-23T15:47:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.990+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7270007420 con 0x7f727c06c0b0 2024-03-23T15:47:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.991+0000 7f72797fa640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f726006b840 0x7f726006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.991+0000 7f727bfff640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f726006b840 0x7f726006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.991+0000 7f727bfff640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f726006b840 0x7f726006dd00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7268036000 tx=0x7f7268008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:48.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:48.995+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f72700580d0 con 0x7f727c06c0b0 2024-03-23T15:47:49.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.012+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7270054370 con 0x7f727c06c0b0 2024-03-23T15:47:49.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.205+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f727c1f1000 con 0x7f727c06c0b0 2024-03-23T15:47:49.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.286+0000 7f72797fa640 1 -- 172.21.15.129:0/441686944 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7f727c1f1000 con 0x7f727c06c0b0 2024-03-23T15:47:49.291 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:49.291 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f726006b840 msgr2=0x7f726006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f726006b840 0x7f726006dd00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7268036000 tx=0x7f7268008030 comp rx=0 tx=0).stop 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 msgr2=0x7f727c0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 0x7f727c0b3e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f727000c400 tx=0x7f727000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 shutdown_connections 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f726006b840 0x7f726006dd00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f727c15a2e0 0x7f727c1f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f727c1544b0 0x7f727c0b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 --2- 172.21.15.129:0/441686944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f727c06c0b0 0x7f727c0b3e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 >> 172.21.15.129:0/441686944 conn(0x7f727c061b50 msgr2=0x7f727c0c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:49.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 shutdown_connections 2024-03-23T15:47:49.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.289+0000 7f7282df4640 1 -- 172.21.15.129:0/441686944 wait complete. 2024-03-23T15:47:49.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T15:47:11.061293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T15:47:24.638536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T15:47:25.617795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T15:47:49.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T15:47:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.405+0000 7f4f7f2a4640 1 Processor -- start 2024-03-23T15:47:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 -- start start 2024-03-23T15:47:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815d4f0 0x7f4f7806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4f7815b940 0x7f4f7815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f7806d570 0x7f4f780cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f4f7806cb20 con 0x7f4f7806d570 2024-03-23T15:47:49.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4f78154c90 con 0x7f4f7815b940 2024-03-23T15:47:49.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.406+0000 7f4f7f2a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7806bb90 con 0x7f4f7815d4f0 2024-03-23T15:47:49.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.407+0000 7f4f7c83f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4f7815b940 0x7f4f7815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.407+0000 7f4f7d841640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f7806d570 0x7f4f780cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.407+0000 7f4f7d040640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815d4f0 0x7f4f7806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.407+0000 7f4f7d841640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f7806d570 0x7f4f780cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60868/0 (socket says 172.21.15.129:60868) 2024-03-23T15:47:49.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.407+0000 7f4f7d841640 1 -- 172.21.15.129:0/3867983910 learned_addr learned my addr 172.21.15.129:0/3867983910 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:49.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.410+0000 7f4f7d841640 1 -- 172.21.15.129:0/3867983910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815d4f0 msgr2=0x7f4f7806c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:49.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.410+0000 7f4f7d841640 1 --2- 172.21.15.129:0/3867983910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815d4f0 0x7f4f7806c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.410+0000 7f4f7d841640 1 -- 172.21.15.129:0/3867983910 >> v2:172.21.15.157:3300/0 conn(0x7f4f7815b940 msgr2=0x7f4f7815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:49.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.410+0000 7f4f7d841640 1 --2- 172.21.15.129:0/3867983910 >> v2:172.21.15.157:3300/0 conn(0x7f4f7815b940 0x7f4f7815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.410+0000 7f4f7d841640 1 -- 172.21.15.129:0/3867983910 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7806c150 con 0x7f4f7806d570 2024-03-23T15:47:49.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.412+0000 7f4f7d841640 1 --2- 172.21.15.129:0/3867983910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780cc290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4f780b26b0 tx=0x7f4f7002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef37f2c6c76d95a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:49.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.415+0000 7f4f6ffff640 1 -- 172.21.15.129:0/3867983910 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7002cb00 con 0x7f4f7806d570 2024-03-23T15:47:49.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.415+0000 7f4f6ffff640 1 -- 172.21.15.129:0/3867983910 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7002cc60 con 0x7f4f7806d570 2024-03-23T15:47:49.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.415+0000 7f4f6ffff640 1 -- 172.21.15.129:0/3867983910 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f700343f0 con 0x7f4f7806d570 2024-03-23T15:47:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/3867983910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 msgr2=0x7f4f780cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/3867983910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780cc290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4f780b26b0 tx=0x7f4f7002bf60 comp rx=0 tx=0).stop 2024-03-23T15:47:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/3867983910 shutdown_connections 2024-03-23T15:47:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/3867983910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780cc290 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/3867983910 >> v2:172.21.15.157:3300/0 conn(0x7f4f7815b940 0x7f4f7815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/3867983910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815d4f0 0x7f4f7806c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.416+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/3867983910 >> 172.21.15.129:0/3867983910 conn(0x7f4f780cfac0 msgr2=0x7f4f780cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:49.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.417+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/3867983910 shutdown_connections 2024-03-23T15:47:49.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.417+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/3867983910 wait complete. 2024-03-23T15:47:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.418+0000 7f4f7f2a4640 1 Processor -- start 2024-03-23T15:47:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.418+0000 7f4f7f2a4640 1 -- start start 2024-03-23T15:47:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.419+0000 7f4f7f2a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7f2a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815b940 0x7f4f780c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7f2a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f7815d4f0 0x7f4f780c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7f2a4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f780ac0a0 con 0x7f4f7806d570 2024-03-23T15:47:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7f2a4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781f2e30 con 0x7f4f7815d4f0 2024-03-23T15:47:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7f2a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f781f3dd0 con 0x7f4f7815b940 2024-03-23T15:47:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7d040640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7d841640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f7815d4f0 0x7f4f780c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7c83f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815b940 0x7f4f780c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7d040640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60872/0 (socket says 172.21.15.129:60872) 2024-03-23T15:47:49.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.420+0000 7f4f7d040640 1 -- 172.21.15.129:0/1767986102 learned_addr learned my addr 172.21.15.129:0/1767986102 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.424+0000 7f4f7d040640 1 -- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815b940 msgr2=0x7f4f780c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.424+0000 7f4f7d040640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815b940 0x7f4f780c59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.424+0000 7f4f7d040640 1 -- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f7815d4f0 msgr2=0x7f4f780c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.424+0000 7f4f7d040640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f7815d4f0 0x7f4f780c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.424+0000 7f4f7d040640 1 -- 172.21.15.129:0/1767986102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f70009b30 con 0x7f4f7806d570 2024-03-23T15:47:49.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.426+0000 7f4f7d040640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780af370 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400e870 tx=0x7f4f7400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:49.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.429+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f740094f0 con 0x7f4f7806d570 2024-03-23T15:47:49.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.429+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7400c940 con 0x7f4f7806d570 2024-03-23T15:47:49.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.429+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7400ccf0 con 0x7f4f7806d570 2024-03-23T15:47:49.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.429+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f780ab720 con 0x7f4f7806d570 2024-03-23T15:47:49.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.430+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f780ab440 con 0x7f4f7806d570 2024-03-23T15:47:49.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.431+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f78075740 con 0x7f4f7806d570 2024-03-23T15:47:49.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.453+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f74013470 con 0x7f4f7806d570 2024-03-23T15:47:49.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.454+0000 7f4f6dffb640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f5c06b860 0x7f4f5c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:49.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.454+0000 7f4f7c83f640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f5c06b860 0x7f4f5c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:49.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.455+0000 7f4f7c83f640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f5c06b860 0x7f4f5c06dd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f781ea1a0 tx=0x7f4f68009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:49.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.457+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4f74059280 con 0x7f4f7806d570 2024-03-23T15:47:49.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.756+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f74055520 con 0x7f4f7806d570 2024-03-23T15:47:49.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.935+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f7806df90 con 0x7f4f7806d570 2024-03-23T15:47:49.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.941+0000 7f4f6dffb640 1 -- 172.21.15.129:0/1767986102 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f4f740210a0 con 0x7f4f7806d570 2024-03-23T15:47:49.945 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f5c06b860 msgr2=0x7f4f5c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f5c06b860 0x7f4f5c06dd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f781ea1a0 tx=0x7f4f68009190 comp rx=0 tx=0).stop 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 msgr2=0x7f4f780af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780af370 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400e870 tx=0x7f4f7400ed30 comp rx=0 tx=0).stop 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 shutdown_connections 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f5c06b860 0x7f4f5c06dd20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f7815d4f0 0x7f4f780c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f7815b940 0x7f4f780c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 --2- 172.21.15.129:0/1767986102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f7806d570 0x7f4f780af370 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.943+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 >> 172.21.15.129:0/1767986102 conn(0x7f4f780cfac0 msgr2=0x7f4f780ca590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.944+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 shutdown_connections 2024-03-23T15:47:49.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:49.944+0000 7f4f7f2a4640 1 -- 172.21.15.129:0/1767986102 wait complete. 2024-03-23T15:47:49.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T15:47:50.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.059+0000 7fe29796c640 1 Processor -- start 2024-03-23T15:47:50.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.059+0000 7fe29796c640 1 -- start start 2024-03-23T15:47:50.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.060+0000 7fe29796c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe29015b230 0x7fe2900cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.060+0000 7fe29796c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe29006dab0 0x7fe2900ca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.060+0000 7fe29796c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2900cee80 con 0x7fe29006d6d0 2024-03-23T15:47:50.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.060+0000 7fe29796c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe2900ccf90 con 0x7fe29015b230 2024-03-23T15:47:50.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.060+0000 7fe29796c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe29015a540 con 0x7fe29006dab0 2024-03-23T15:47:50.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.061+0000 7fe294f07640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe29015b230 0x7fe2900cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.061+0000 7fe295f09640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe29006dab0 0x7fe2900ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.061+0000 7fe294f07640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe29015b230 0x7fe2900cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60880/0 (socket says 172.21.15.129:60880) 2024-03-23T15:47:50.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.061+0000 7fe294f07640 1 -- 172.21.15.129:0/1008119692 learned_addr learned my addr 172.21.15.129:0/1008119692 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.063+0000 7fe294f07640 1 -- 172.21.15.129:0/1008119692 >> v2:172.21.15.157:3300/0 conn(0x7fe29006dab0 msgr2=0x7fe2900ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.063+0000 7fe294f07640 1 --2- 172.21.15.129:0/1008119692 >> v2:172.21.15.157:3300/0 conn(0x7fe29006dab0 0x7fe2900ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.063+0000 7fe294f07640 1 -- 172.21.15.129:0/1008119692 >> v1:172.21.15.157:6789/0 conn(0x7fe29006d6d0 legacy=0x7fe29006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.063+0000 7fe294f07640 1 -- 172.21.15.129:0/1008119692 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2900cc710 con 0x7fe29015b230 2024-03-23T15:47:50.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.065+0000 7fe294f07640 1 --2- 172.21.15.129:0/1008119692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2900cf210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe29006cfb0 tx=0x7fe28002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84b239338f093f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.069+0000 7fe287fff640 1 -- 172.21.15.129:0/1008119692 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28002f850 con 0x7fe29015b230 2024-03-23T15:47:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.069+0000 7fe287fff640 1 -- 172.21.15.129:0/1008119692 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe28002f9b0 con 0x7fe29015b230 2024-03-23T15:47:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.069+0000 7fe287fff640 1 -- 172.21.15.129:0/1008119692 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28002fd40 con 0x7fe29015b230 2024-03-23T15:47:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 -- 172.21.15.129:0/1008119692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 msgr2=0x7fe2900cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 --2- 172.21.15.129:0/1008119692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2900cf210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe29006cfb0 tx=0x7fe28002ba30 comp rx=0 tx=0).stop 2024-03-23T15:47:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 -- 172.21.15.129:0/1008119692 shutdown_connections 2024-03-23T15:47:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 --2- 172.21.15.129:0/1008119692 >> v2:172.21.15.157:3300/0 conn(0x7fe29006dab0 0x7fe2900ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 --2- 172.21.15.129:0/1008119692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2900cf210 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 -- 172.21.15.129:0/1008119692 >> 172.21.15.129:0/1008119692 conn(0x7fe2900cfac0 msgr2=0x7fe2900cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 -- 172.21.15.129:0/1008119692 shutdown_connections 2024-03-23T15:47:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.070+0000 7fe29796c640 1 -- 172.21.15.129:0/1008119692 wait complete. 2024-03-23T15:47:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.072+0000 7fe29796c640 1 Processor -- start 2024-03-23T15:47:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.072+0000 7fe29796c640 1 -- start start 2024-03-23T15:47:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.073+0000 7fe29796c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe29006d6d0 0x7fe290156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.073+0000 7fe29796c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe29006dab0 0x7fe2901f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.073+0000 7fe29796c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2901f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.073+0000 7fe29796c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe2901f0c70 con 0x7fe29015b230 2024-03-23T15:47:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.073+0000 7fe29796c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2900c55f0 con 0x7fe29006dab0 2024-03-23T15:47:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.073+0000 7fe29796c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2900c58b0 con 0x7fe29006d6d0 2024-03-23T15:47:50.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.074+0000 7fe295f09640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2901f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.074+0000 7fe295708640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe29006d6d0 0x7fe290156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.074+0000 7fe294f07640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe29006dab0 0x7fe2901f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.074+0000 7fe295708640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe29006d6d0 0x7fe290156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49012/0 (socket says 172.21.15.129:49012) 2024-03-23T15:47:50.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.074+0000 7fe295708640 1 -- 172.21.15.129:0/1198889122 learned_addr learned my addr 172.21.15.129:0/1198889122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:50.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.078+0000 7fe294f07640 1 -- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe29006d6d0 msgr2=0x7fe290156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.078+0000 7fe294f07640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe29006d6d0 0x7fe290156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.078+0000 7fe294f07640 1 -- 172.21.15.129:0/1198889122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 msgr2=0x7fe2901f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.078+0000 7fe294f07640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2901f0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.078+0000 7fe294f07640 1 -- 172.21.15.129:0/1198889122 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2800094d0 con 0x7fe29006dab0 2024-03-23T15:47:50.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.080+0000 7fe294f07640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe29006dab0 0x7fe2901f3e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe280033000 tx=0x7fe280004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:50.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.086+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280037c40 con 0x7fe29006dab0 2024-03-23T15:47:50.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.086+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe280037da0 con 0x7fe29006dab0 2024-03-23T15:47:50.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.086+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2900c5b90 con 0x7fe29006dab0 2024-03-23T15:47:50.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.086+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280041640 con 0x7fe29006dab0 2024-03-23T15:47:50.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.086+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2901f3600 con 0x7fe29006dab0 2024-03-23T15:47:50.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.088+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe290075740 con 0x7fe29006dab0 2024-03-23T15:47:50.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.394+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe2800417a0 con 0x7fe29006dab0 2024-03-23T15:47:50.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.394+0000 7fe285ffb640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe27406b890 0x7fe27406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.394+0000 7fe295708640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe27406b890 0x7fe27406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.395+0000 7fe295708640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe27406b890 0x7fe27406dd50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0017a0 tx=0x7fe28c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:50.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.397+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe28003c070 con 0x7fe29006dab0 2024-03-23T15:47:50.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.423+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe280078490 con 0x7fe29006dab0 2024-03-23T15:47:50.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.692+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe2900c6360 con 0x7fe29006dab0 2024-03-23T15:47:50.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.703+0000 7fe285ffb640 1 -- 172.21.15.129:0/1198889122 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fe28007bb50 con 0x7fe29006dab0 2024-03-23T15:47:50.707 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-03-23T15:47:50.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe27406b890 msgr2=0x7fe27406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe27406b890 0x7fe27406dd50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0017a0 tx=0x7fe28c009040 comp rx=0 tx=0).stop 2024-03-23T15:47:50.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe29006dab0 msgr2=0x7fe2901f3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe29006dab0 0x7fe2901f3e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe280033000 tx=0x7fe280004730 comp rx=0 tx=0).stop 2024-03-23T15:47:50.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 shutdown_connections 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe29015b230 0x7fe2901f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe27406b890 0x7fe27406dd50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe29006dab0 0x7fe2901f3e00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 --2- 172.21.15.129:0/1198889122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe29006d6d0 0x7fe290156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 >> 172.21.15.129:0/1198889122 conn(0x7fe2900cfac0 msgr2=0x7fe290159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 shutdown_connections 2024-03-23T15:47:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.706+0000 7fe29796c640 1 -- 172.21.15.129:0/1198889122 wait complete. 2024-03-23T15:47:50.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T15:47:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.829+0000 7f866f68e640 1 Processor -- start 2024-03-23T15:47:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.830+0000 7f866f68e640 1 -- start start 2024-03-23T15:47:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.831+0000 7f866f68e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f866806d570 0x7f866806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.831+0000 7f866f68e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8668154c90 con 0x7f866815d4f0 2024-03-23T15:47:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.831+0000 7f866f68e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f866806bb90 con 0x7f866815b940 2024-03-23T15:47:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.831+0000 7f866f68e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f866806cb20 con 0x7f866806d570 2024-03-23T15:47:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.831+0000 7f866d42a640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f866815d4f0 0x7f866815a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43082/0 (socket says 172.21.15.129:43082) 2024-03-23T15:47:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.831+0000 7f866d42a640 1 -- 172.21.15.129:0/354711441 learned_addr learned my addr 172.21.15.129:0/354711441 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.832+0000 7f866dc2b640 1 --2- 172.21.15.129:0/354711441 >> v2:172.21.15.157:3300/0 conn(0x7f866806d570 0x7f866806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.834+0000 7f865ffff640 1 -- 172.21.15.129:0/354711441 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1834693716 0 0) 0x7f8668154c90 con 0x7f866815d4f0 2024-03-23T15:47:50.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.834+0000 7f865ffff640 1 -- 172.21.15.129:0/354711441 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8654003520 con 0x7f866815d4f0 2024-03-23T15:47:50.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.836+0000 7f866dc2b640 1 -- 172.21.15.129:0/354711441 >> v1:172.21.15.157:6789/0 conn(0x7f866815b940 legacy=0x7f86680cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.836+0000 7f866dc2b640 1 -- 172.21.15.129:0/354711441 >> v1:172.21.15.129:6789/0 conn(0x7f866815d4f0 legacy=0x7f866815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.836+0000 7f866dc2b640 1 -- 172.21.15.129:0/354711441 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866806c150 con 0x7f866806d570 2024-03-23T15:47:50.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.842+0000 7f866dc2b640 1 --2- 172.21.15.129:0/354711441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f866806c5e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8660009d90 tx=0x7f866002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bed446d3c99dc637 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:50.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.847+0000 7f865ffff640 1 -- 172.21.15.129:0/354711441 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866002cd80 con 0x7f866806d570 2024-03-23T15:47:50.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.847+0000 7f865ffff640 1 -- 172.21.15.129:0/354711441 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8660034440 con 0x7f866806d570 2024-03-23T15:47:50.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.847+0000 7f865ffff640 1 -- 172.21.15.129:0/354711441 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600347d0 con 0x7f866806d570 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 -- 172.21.15.129:0/354711441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 msgr2=0x7f866806c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 --2- 172.21.15.129:0/354711441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f866806c5e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8660009d90 tx=0x7f866002bfd0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 -- 172.21.15.129:0/354711441 shutdown_connections 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 --2- 172.21.15.129:0/354711441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f866806c5e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 -- 172.21.15.129:0/354711441 >> 172.21.15.129:0/354711441 conn(0x7f86680cfac0 msgr2=0x7f86680cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 -- 172.21.15.129:0/354711441 shutdown_connections 2024-03-23T15:47:50.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.848+0000 7f866f68e640 1 -- 172.21.15.129:0/354711441 wait complete. 2024-03-23T15:47:50.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.849+0000 7f866f68e640 1 Processor -- start 2024-03-23T15:47:50.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.849+0000 7f866f68e640 1 -- start start 2024-03-23T15:47:50.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866f68e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f86681f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866f68e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f866815b940 0x7f86680b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866f68e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 0x7f866815a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866f68e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f86680c36d0 con 0x7f866815d4f0 2024-03-23T15:47:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866f68e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f86681efa00 con 0x7f866806d570 2024-03-23T15:47:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866f68e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f86681f3ec0 con 0x7f866815b940 2024-03-23T15:47:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866dc2b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 0x7f866815a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.850+0000 7f866d42a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f86681f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.851+0000 7f866cc29640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f866815b940 0x7f86680b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.851+0000 7f866dc2b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 0x7f866815a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54266/0 (socket says 172.21.15.129:54266) 2024-03-23T15:47:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.851+0000 7f866dc2b640 1 -- 172.21.15.129:0/1804316484 learned_addr learned my addr 172.21.15.129:0/1804316484 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.856+0000 7f866dc2b640 1 -- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f866815b940 msgr2=0x7f86680b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.856+0000 7f866dc2b640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f866815b940 0x7f86680b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.856+0000 7f866dc2b640 1 -- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 msgr2=0x7f86681f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.856+0000 7f866dc2b640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f86681f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.856+0000 7f866dc2b640 1 -- 172.21.15.129:0/1804316484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660009a70 con 0x7f866815d4f0 2024-03-23T15:47:50.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.858+0000 7f866dc2b640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 0x7f866815a210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8660009eb0 tx=0x7f8660004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:50.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.863+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660034df0 con 0x7f866815d4f0 2024-03-23T15:47:50.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.863+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f866003f420 con 0x7f866815d4f0 2024-03-23T15:47:50.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.863+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866003f7b0 con 0x7f866815d4f0 2024-03-23T15:47:50.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.863+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86680b0e70 con 0x7f866815d4f0 2024-03-23T15:47:50.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.863+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86680afaf0 con 0x7f866815d4f0 2024-03-23T15:47:50.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.864+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8668075740 con 0x7f866815d4f0 2024-03-23T15:47:50.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.894+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f866003f910 con 0x7f866815d4f0 2024-03-23T15:47:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.894+0000 7f865dffb640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f864c06b860 0x7f864c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.894+0000 7f866d42a640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f864c06b860 0x7f864c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.895+0000 7f866d42a640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f864c06b860 0x7f864c06dd20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8664005880 tx=0x7f8664009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:50.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:50.899+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f866007b9f0 con 0x7f866815d4f0 2024-03-23T15:47:51.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.151+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8660077c90 con 0x7f866815d4f0 2024-03-23T15:47:51.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.335+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f86681594e0 con 0x7f866815d4f0 2024-03-23T15:47:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.341+0000 7f865dffb640 1 -- 172.21.15.129:0/1804316484 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f866004d020 con 0x7f866815d4f0 2024-03-23T15:47:51.344 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f864c06b860 msgr2=0x7f864c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f864c06b860 0x7f864c06dd20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8664005880 tx=0x7f8664009040 comp rx=0 tx=0).stop 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 msgr2=0x7f866815a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 0x7f866815a210 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8660009eb0 tx=0x7f8660004a40 comp rx=0 tx=0).stop 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 shutdown_connections 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f866815d4f0 0x7f866815a210 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f866815b940 0x7f86680b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f864c06b860 0x7f864c06dd20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 --2- 172.21.15.129:0/1804316484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f866806d570 0x7f86681f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 >> 172.21.15.129:0/1804316484 conn(0x7f86680cfac0 msgr2=0x7f8668158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:51.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 shutdown_connections 2024-03-23T15:47:51.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.344+0000 7f866f68e640 1 -- 172.21.15.129:0/1804316484 wait complete. 2024-03-23T15:47:51.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:51.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.458+0000 7eff691b6640 1 Processor -- start 2024-03-23T15:47:51.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.458+0000 7eff691b6640 1 -- start start 2024-03-23T15:47:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.459+0000 7eff691b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff6406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.459+0000 7eff691b6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7eff6406c0b0 0x7eff640c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.459+0000 7eff691b6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff640cbcb0 con 0x7eff641544b0 2024-03-23T15:47:51.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.459+0000 7eff691b6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7eff640c5be0 con 0x7eff6406c0b0 2024-03-23T15:47:51.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.459+0000 7eff691b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7eff6415b280 con 0x7eff6415a2e0 2024-03-23T15:47:51.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.460+0000 7eff62d9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7eff641544b0 0x7eff640caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43084/0 (socket says 172.21.15.129:43084) 2024-03-23T15:47:51.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.460+0000 7eff62d9d640 1 -- 172.21.15.129:0/2291654999 learned_addr learned my addr 172.21.15.129:0/2291654999 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:51.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.460+0000 7eff6259c640 1 --2- 172.21.15.129:0/2291654999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff6406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:51.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.460+0000 7eff6359e640 1 --2- 172.21.15.129:0/2291654999 >> v2:172.21.15.157:3300/0 conn(0x7eff6406c0b0 0x7eff640c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.463+0000 7eff61d9b640 1 -- 172.21.15.129:0/2291654999 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302927429 0 0) 0x7eff640cbcb0 con 0x7eff641544b0 2024-03-23T15:47:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.463+0000 7eff61d9b640 1 -- 172.21.15.129:0/2291654999 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff4c003520 con 0x7eff641544b0 2024-03-23T15:47:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.463+0000 7eff6259c640 1 -- 172.21.15.129:0/2291654999 >> v2:172.21.15.157:3300/0 conn(0x7eff6406c0b0 msgr2=0x7eff640c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.463+0000 7eff6259c640 1 --2- 172.21.15.129:0/2291654999 >> v2:172.21.15.157:3300/0 conn(0x7eff6406c0b0 0x7eff640c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.463+0000 7eff6259c640 1 -- 172.21.15.129:0/2291654999 >> v1:172.21.15.129:6789/0 conn(0x7eff641544b0 legacy=0x7eff640caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.463+0000 7eff6259c640 1 -- 172.21.15.129:0/2291654999 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6415a9e0 con 0x7eff6415a2e0 2024-03-23T15:47:51.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.466+0000 7eff6259c640 1 --2- 172.21.15.129:0/2291654999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff6406bb70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7eff580097a0 tx=0x7eff5802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f396d3fb9816c532 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:51.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.470+0000 7eff61d9b640 1 -- 172.21.15.129:0/2291654999 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58030610 con 0x7eff6415a2e0 2024-03-23T15:47:51.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.470+0000 7eff61d9b640 1 -- 172.21.15.129:0/2291654999 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff58030770 con 0x7eff6415a2e0 2024-03-23T15:47:51.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.470+0000 7eff61d9b640 1 -- 172.21.15.129:0/2291654999 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58030b00 con 0x7eff6415a2e0 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 -- 172.21.15.129:0/2291654999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 msgr2=0x7eff6406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 --2- 172.21.15.129:0/2291654999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff6406bb70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7eff580097a0 tx=0x7eff5802b960 comp rx=0 tx=0).stop 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 -- 172.21.15.129:0/2291654999 shutdown_connections 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 --2- 172.21.15.129:0/2291654999 >> v2:172.21.15.157:3300/0 conn(0x7eff6406c0b0 0x7eff640c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 --2- 172.21.15.129:0/2291654999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff6406bb70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 -- 172.21.15.129:0/2291654999 >> 172.21.15.129:0/2291654999 conn(0x7eff64061b50 msgr2=0x7eff64061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 -- 172.21.15.129:0/2291654999 shutdown_connections 2024-03-23T15:47:51.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.471+0000 7eff691b6640 1 -- 172.21.15.129:0/2291654999 wait complete. 2024-03-23T15:47:51.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.472+0000 7eff691b6640 1 Processor -- start 2024-03-23T15:47:51.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.472+0000 7eff691b6640 1 -- start start 2024-03-23T15:47:51.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff691b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 0x7eff6414a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:51.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff691b6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff641544b0 0x7eff640cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:51.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff691b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff640ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:51.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff691b6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff640ce160 con 0x7eff6406c0b0 2024-03-23T15:47:51.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff691b6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7eff640cdae0 con 0x7eff641544b0 2024-03-23T15:47:51.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff691b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7eff641f4710 con 0x7eff6415a2e0 2024-03-23T15:47:51.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff62d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 0x7eff6414a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:51.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.473+0000 7eff6359e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff640ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:51.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.474+0000 7eff6259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff641544b0 0x7eff640cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:51.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.474+0000 7eff62d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 0x7eff6414a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54270/0 (socket says 172.21.15.129:54270) 2024-03-23T15:47:51.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.474+0000 7eff62d9d640 1 -- 172.21.15.129:0/2793401721 learned_addr learned my addr 172.21.15.129:0/2793401721 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:51.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.477+0000 7eff62d9d640 1 -- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 msgr2=0x7eff640ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:51.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.477+0000 7eff62d9d640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff640ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.477+0000 7eff62d9d640 1 -- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff641544b0 msgr2=0x7eff640cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:51.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.477+0000 7eff62d9d640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff641544b0 0x7eff640cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:51.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.477+0000 7eff62d9d640 1 -- 172.21.15.129:0/2793401721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff58009400 con 0x7eff6406c0b0 2024-03-23T15:47:51.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.479+0000 7eff62d9d640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 0x7eff6414a6a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff5400c430 tx=0x7eff5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:51.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.483+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54005ce0 con 0x7eff6406c0b0 2024-03-23T15:47:51.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.483+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff54005e40 con 0x7eff6406c0b0 2024-03-23T15:47:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.483+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff640cdd80 con 0x7eff6406c0b0 2024-03-23T15:47:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.483+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff540072f0 con 0x7eff6406c0b0 2024-03-23T15:47:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.484+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff64154d50 con 0x7eff6406c0b0 2024-03-23T15:47:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.485+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff640ca770 con 0x7eff6406c0b0 2024-03-23T15:47:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.507+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff54007450 con 0x7eff6406c0b0 2024-03-23T15:47:51.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.507+0000 7eff4b7fe640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7eff5006b810 0x7eff5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:51.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.508+0000 7eff6259c640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7eff5006b810 0x7eff5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:51.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.508+0000 7eff6259c640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7eff5006b810 0x7eff5006dcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7eff58036000 tx=0x7eff58008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:51.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.510+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff54058480 con 0x7eff6406c0b0 2024-03-23T15:47:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:51.799+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff54054720 con 0x7eff6406c0b0 2024-03-23T15:47:52.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.017+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff640ce2a0 con 0x7eff6406c0b0 2024-03-23T15:47:52.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.031+0000 7eff4b7fe640 1 -- 172.21.15.129:0/2793401721 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6920 (secure 0 0 0) 0x7eff640ce2a0 con 0x7eff6406c0b0 2024-03-23T15:47:52.034 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:52.034 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4234,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:47:40.883381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4250,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6837/3877606926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3877606926},{"type":"v1","addr":"172.21.15.157:6837","nonce":3877606926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4250,4253]},"id":1}]} 2024-03-23T15:47:52.035 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.033+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7eff5006b810 msgr2=0x7eff5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.033+0000 7eff691b6640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7eff5006b810 0x7eff5006dcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7eff58036000 tx=0x7eff58008030 comp rx=0 tx=0).stop 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.033+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 msgr2=0x7eff6414a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.033+0000 7eff691b6640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 0x7eff6414a6a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff5400c430 tx=0x7eff5400c8f0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 shutdown_connections 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6415a2e0 0x7eff640ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7eff5006b810 0x7eff5006dcd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff641544b0 0x7eff640cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 --2- 172.21.15.129:0/2793401721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff6406c0b0 0x7eff6414a6a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 >> 172.21.15.129:0/2793401721 conn(0x7eff64061b50 msgr2=0x7eff640c6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 shutdown_connections 2024-03-23T15:47:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.034+0000 7eff691b6640 1 -- 172.21.15.129:0/2793401721 wait complete. 2024-03-23T15:47:52.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T15:47:52.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.146+0000 7fdb0950c640 1 Processor -- start 2024-03-23T15:47:52.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.147+0000 7fdb0950c640 1 -- start start 2024-03-23T15:47:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.148+0000 7fdb0950c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb0415a2e0 0x7fdb0406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.148+0000 7fdb0950c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdb0406c0b0 0x7fdb040c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.148+0000 7fdb0950c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb040cbcb0 con 0x7fdb041544b0 2024-03-23T15:47:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.148+0000 7fdb0950c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdb040c5be0 con 0x7fdb0406c0b0 2024-03-23T15:47:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.148+0000 7fdb0950c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0415b280 con 0x7fdb0415a2e0 2024-03-23T15:47:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.149+0000 7fdb027fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb0415a2e0 0x7fdb0406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.149+0000 7fdb02ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fdb041544b0 0x7fdb040caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43098/0 (socket says 172.21.15.129:43098) 2024-03-23T15:47:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.149+0000 7fdb02ffd640 1 -- 172.21.15.129:0/1196615151 learned_addr learned my addr 172.21.15.129:0/1196615151 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.149+0000 7fdb037fe640 1 --2- 172.21.15.129:0/1196615151 >> v2:172.21.15.157:3300/0 conn(0x7fdb0406c0b0 0x7fdb040c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.151+0000 7fdb027fc640 1 -- 172.21.15.129:0/1196615151 >> v2:172.21.15.157:3300/0 conn(0x7fdb0406c0b0 msgr2=0x7fdb040c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:52.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.151+0000 7fdb027fc640 1 --2- 172.21.15.129:0/1196615151 >> v2:172.21.15.157:3300/0 conn(0x7fdb0406c0b0 0x7fdb040c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.151+0000 7fdb027fc640 1 -- 172.21.15.129:0/1196615151 >> v1:172.21.15.129:6789/0 conn(0x7fdb041544b0 legacy=0x7fdb040caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.151+0000 7fdb027fc640 1 -- 172.21.15.129:0/1196615151 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0415a9e0 con 0x7fdb0415a2e0 2024-03-23T15:47:52.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.153+0000 7fdb027fc640 1 --2- 172.21.15.129:0/1196615151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb0415a2e0 0x7fdb0406bb70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80097a0 tx=0x7fdaf802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265d87ed73987dce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.155+0000 7fdb01ffb640 1 -- 172.21.15.129:0/1196615151 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8030610 con 0x7fdb0415a2e0 2024-03-23T15:47:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb01ffb640 1 -- 172.21.15.129:0/1196615151 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf8030770 con 0x7fdb0415a2e0 2024-03-23T15:47:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb01ffb640 1 -- 172.21.15.129:0/1196615151 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8030b00 con 0x7fdb0415a2e0 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 -- 172.21.15.129:0/1196615151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb0415a2e0 msgr2=0x7fdb0406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 --2- 172.21.15.129:0/1196615151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb0415a2e0 0x7fdb0406bb70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80097a0 tx=0x7fdaf802b960 comp rx=0 tx=0).stop 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 -- 172.21.15.129:0/1196615151 shutdown_connections 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 --2- 172.21.15.129:0/1196615151 >> v2:172.21.15.157:3300/0 conn(0x7fdb0406c0b0 0x7fdb040c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 --2- 172.21.15.129:0/1196615151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb0415a2e0 0x7fdb0406bb70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 -- 172.21.15.129:0/1196615151 >> 172.21.15.129:0/1196615151 conn(0x7fdb04061b50 msgr2=0x7fdb04061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 -- 172.21.15.129:0/1196615151 shutdown_connections 2024-03-23T15:47:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.158+0000 7fdb0950c640 1 -- 172.21.15.129:0/1196615151 wait complete. 2024-03-23T15:47:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.159+0000 7fdb0950c640 1 Processor -- start 2024-03-23T15:47:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 -- start start 2024-03-23T15:47:52.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb0406c0b0 0x7fdb041f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb041544b0 0x7fdb04155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb0415a2e0 0x7fdb04155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04154c50 con 0x7fdb0415a2e0 2024-03-23T15:47:52.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdb041f4760 con 0x7fdb0406c0b0 2024-03-23T15:47:52.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.160+0000 7fdb0950c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb04157770 con 0x7fdb041544b0 2024-03-23T15:47:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.161+0000 7fdb027fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb041544b0 0x7fdb04155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.161+0000 7fdb037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb0415a2e0 0x7fdb04155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.161+0000 7fdb02ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb0406c0b0 0x7fdb041f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.161+0000 7fdb027fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb041544b0 0x7fdb04155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35836/0 (socket says 172.21.15.129:35836) 2024-03-23T15:47:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.161+0000 7fdb027fc640 1 -- 172.21.15.129:0/865768325 learned_addr learned my addr 172.21.15.129:0/865768325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.165+0000 7fdb037fe640 1 -- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb041544b0 msgr2=0x7fdb04155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.165+0000 7fdb037fe640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb041544b0 0x7fdb04155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.165+0000 7fdb037fe640 1 -- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb0406c0b0 msgr2=0x7fdb041f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.165+0000 7fdb037fe640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb0406c0b0 0x7fdb041f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.165+0000 7fdb037fe640 1 -- 172.21.15.129:0/865768325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf8009400 con 0x7fdb0415a2e0 2024-03-23T15:47:52.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.167+0000 7fdb037fe640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb0415a2e0 0x7fdb04155640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00eba0 tx=0x7fdafc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:52.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.171+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc00cd20 con 0x7fdb0415a2e0 2024-03-23T15:47:52.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.171+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdafc00f040 con 0x7fdb0415a2e0 2024-03-23T15:47:52.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.171+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb040cdb10 con 0x7fdb0415a2e0 2024-03-23T15:47:52.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.171+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0146c0 con 0x7fdb0415a2e0 2024-03-23T15:47:52.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.171+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0414e3d0 con 0x7fdb0415a2e0 2024-03-23T15:47:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.172+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb04154e20 con 0x7fdb0415a2e0 2024-03-23T15:47:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.191+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdafc014820 con 0x7fdb0415a2e0 2024-03-23T15:47:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.191+0000 7fdaeb7fe640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf006b860 0x7fdaf006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.192+0000 7fdb02ffd640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf006b860 0x7fdaf006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.192+0000 7fdb02ffd640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf006b860 0x7fdaf006dd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4007fd0 tx=0x7fdaf4007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:52.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.440+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdafc059a20 con 0x7fdb0415a2e0 2024-03-23T15:47:52.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.440+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdafc055cc0 con 0x7fdb0415a2e0 2024-03-23T15:47:52.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.625+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb041575b0 con 0x7fdb0415a2e0 2024-03-23T15:47:52.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.669+0000 7fdaeb7fe640 1 -- 172.21.15.129:0/865768325 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16645 (secure 0 0 0) 0x7fdafc018020 con 0x7fdb0415a2e0 2024-03-23T15:47:52.672 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:52.672 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:47:28.316619+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T15:47:52.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf006b860 msgr2=0x7fdaf006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf006b860 0x7fdaf006dd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4007fd0 tx=0x7fdaf4007e80 comp rx=0 tx=0).stop 2024-03-23T15:47:52.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb0415a2e0 msgr2=0x7fdb04155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb0415a2e0 0x7fdb04155640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdafc00eba0 tx=0x7fdafc00c490 comp rx=0 tx=0).stop 2024-03-23T15:47:52.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 shutdown_connections 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb0415a2e0 0x7fdb04155640 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb041544b0 0x7fdb04155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf006b860 0x7fdaf006dd20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 --2- 172.21.15.129:0/865768325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb0406c0b0 0x7fdb041f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 >> 172.21.15.129:0/865768325 conn(0x7fdb04061b50 msgr2=0x7fdb040b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 shutdown_connections 2024-03-23T15:47:52.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.671+0000 7fdb0950c640 1 -- 172.21.15.129:0/865768325 wait complete. 2024-03-23T15:47:52.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:47:52.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.787+0000 7f73ca27c640 1 Processor -- start 2024-03-23T15:47:52.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.787+0000 7f73ca27c640 1 -- start start 2024-03-23T15:47:52.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.788+0000 7f73ca27c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.788+0000 7f73ca27c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f73c40ceb00 0x7f73c406c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.788+0000 7f73ca27c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73c415b290 con 0x7f73c40cef40 2024-03-23T15:47:52.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.788+0000 7f73ca27c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f73c40ce800 con 0x7f73c40ceb00 2024-03-23T15:47:52.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.788+0000 7f73ca27c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73c4154a70 con 0x7f73c40ccd60 2024-03-23T15:47:52.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.789+0000 7f73c37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f73c40ceb00 0x7f73c406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.789+0000 7f73c3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.789+0000 7f73c8819640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f73c40cef40 0x7f73c415b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:32878/0 (socket says 172.21.15.129:32878) 2024-03-23T15:47:52.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.789+0000 7f73c8819640 1 -- 172.21.15.129:0/1296019333 learned_addr learned my addr 172.21.15.129:0/1296019333 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:47:52.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.792+0000 7f73c3fff640 1 -- 172.21.15.129:0/1296019333 >> v2:172.21.15.129:3300/0 conn(0x7f73c40ceb00 msgr2=0x7f73c406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:52.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.792+0000 7f73c3fff640 1 --2- 172.21.15.129:0/1296019333 >> v2:172.21.15.129:3300/0 conn(0x7f73c40ceb00 0x7f73c406c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.792+0000 7f73c3fff640 1 -- 172.21.15.129:0/1296019333 >> v1:172.21.15.157:6789/0 conn(0x7f73c40cef40 legacy=0x7f73c415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.792+0000 7f73c3fff640 1 -- 172.21.15.129:0/1296019333 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c40cd950 con 0x7f73c40ccd60 2024-03-23T15:47:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.793+0000 7f73c3fff640 1 --2- 172.21.15.129:0/1296019333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c415ad50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73b40097a0 tx=0x7f73b402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=437346cf7c8ff244 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.798+0000 7f73c2ffd640 1 -- 172.21.15.129:0/1296019333 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b4030610 con 0x7f73c40ccd60 2024-03-23T15:47:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.798+0000 7f73c2ffd640 1 -- 172.21.15.129:0/1296019333 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b4030770 con 0x7f73c40ccd60 2024-03-23T15:47:52.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.798+0000 7f73c2ffd640 1 -- 172.21.15.129:0/1296019333 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b4030b00 con 0x7f73c40ccd60 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 -- 172.21.15.129:0/1296019333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 msgr2=0x7f73c415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 --2- 172.21.15.129:0/1296019333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c415ad50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73b40097a0 tx=0x7f73b402b960 comp rx=0 tx=0).stop 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 -- 172.21.15.129:0/1296019333 shutdown_connections 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 --2- 172.21.15.129:0/1296019333 >> v2:172.21.15.129:3300/0 conn(0x7f73c40ceb00 0x7f73c406c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 --2- 172.21.15.129:0/1296019333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c415ad50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 -- 172.21.15.129:0/1296019333 >> 172.21.15.129:0/1296019333 conn(0x7f73c40b1840 msgr2=0x7f73c40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 -- 172.21.15.129:0/1296019333 shutdown_connections 2024-03-23T15:47:52.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.799+0000 7f73ca27c640 1 -- 172.21.15.129:0/1296019333 wait complete. 2024-03-23T15:47:52.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.800+0000 7f73ca27c640 1 Processor -- start 2024-03-23T15:47:52.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 -- start start 2024-03-23T15:47:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c41577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73c40ceb00 0x7f73c415cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c40cef40 0x7f73c40c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73c41f11d0 con 0x7f73c40cef40 2024-03-23T15:47:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73c41f03b0 con 0x7f73c40ceb00 2024-03-23T15:47:52.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.801+0000 7f73ca27c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73c41f0650 con 0x7f73c40ccd60 2024-03-23T15:47:52.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.802+0000 7f73c3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c41577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.802+0000 7f73c37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73c40ceb00 0x7f73c415cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.802+0000 7f73c8819640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c40cef40 0x7f73c40c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.802+0000 7f73c3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c41577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35844/0 (socket says 172.21.15.129:35844) 2024-03-23T15:47:52.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.802+0000 7f73c3fff640 1 -- 172.21.15.129:0/165078116 learned_addr learned my addr 172.21.15.129:0/165078116 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:52.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.806+0000 7f73c3fff640 1 -- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73c40ceb00 msgr2=0x7f73c415cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.806+0000 7f73c3fff640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73c40ceb00 0x7f73c415cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.806+0000 7f73c3fff640 1 -- 172.21.15.129:0/165078116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c40cef40 msgr2=0x7f73c40c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:52.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.806+0000 7f73c3fff640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c40cef40 0x7f73c40c5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:52.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.806+0000 7f73c3fff640 1 -- 172.21.15.129:0/165078116 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b4009400 con 0x7f73c40ccd60 2024-03-23T15:47:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.809+0000 7f73c3fff640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c41577f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009890 tx=0x7f73b4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.813+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b403e040 con 0x7f73c40ccd60 2024-03-23T15:47:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.813+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b4039ce0 con 0x7f73c40ccd60 2024-03-23T15:47:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.813+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c41f0930 con 0x7f73c40ccd60 2024-03-23T15:47:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.813+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b4043680 con 0x7f73c40ccd60 2024-03-23T15:47:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.814+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c41f0ef0 con 0x7f73c40ccd60 2024-03-23T15:47:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.815+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c404f530 con 0x7f73c40ccd60 2024-03-23T15:47:52.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.840+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f73b40423e0 con 0x7f73c40ccd60 2024-03-23T15:47:52.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.840+0000 7f73c0ff9640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b006b840 0x7f73b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:52.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.841+0000 7f73c37fe640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b006b840 0x7f73b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:52.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.841+0000 7f73c37fe640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b006b840 0x7f73b006dd00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f73c415d440 tx=0x7f73b8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:52.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:52.844+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f73b407af80 con 0x7f73c40ccd60 2024-03-23T15:47:53.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.147+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73b4077220 con 0x7f73c40ccd60 2024-03-23T15:47:53.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.371+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73c40686f0 con 0x7f73c40ccd60 2024-03-23T15:47:53.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.390+0000 7f73c0ff9640 1 -- 172.21.15.129:0/165078116 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6920 (secure 0 0 0) 0x7f73b407a8e0 con 0x7f73c40ccd60 2024-03-23T15:47:53.393 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:47:53.393 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4234,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:47:40.883381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4250,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.157:6837/3877606926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3877606926},{"type":"v1","addr":"172.21.15.157:6837","nonce":3877606926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4250,4253]},"id":1}]} 2024-03-23T15:47:53.393 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-03-23T15:47:53.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b006b840 msgr2=0x7f73b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:53.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b006b840 0x7f73b006dd00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f73c415d440 tx=0x7f73b8009190 comp rx=0 tx=0).stop 2024-03-23T15:47:53.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 msgr2=0x7f73c41577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c41577f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73b4009890 tx=0x7f73b4004730 comp rx=0 tx=0).stop 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 shutdown_connections 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c40cef40 0x7f73c40c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b006b840 0x7f73b006dd00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73c40ceb00 0x7f73c415cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 --2- 172.21.15.129:0/165078116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73c40ccd60 0x7f73c41577f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 >> 172.21.15.129:0/165078116 conn(0x7f73c40b1840 msgr2=0x7f73c40cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 shutdown_connections 2024-03-23T15:47:53.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:53.392+0000 7f73ca27c640 1 -- 172.21.15.129:0/165078116 wait complete. 2024-03-23T15:47:54.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:47:40.883381+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4250, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4250': {'gid': 4250, 'name': 'f', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.157:6837/3877606926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6836', 'nonce': 3877606926}, {'type': 'v1', 'addr': '172.21.15.157:6837', 'nonce': 3877606926}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4250, 4253]} 2024-03-23T15:47:54.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-23T15:47:54.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T15:47:54.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.112+0000 7f12a33a2640 1 Processor -- start 2024-03-23T15:47:54.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.112+0000 7f12a33a2640 1 -- start start 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.113+0000 7f12a33a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.113+0000 7f12a33a2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f129c15dcf0 0x7f129c06bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.113+0000 7f12a33a2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f129c0cb5c0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.113+0000 7f12a33a2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f129c0cabe0 con 0x7f129c15dcf0 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.113+0000 7f12a33a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f129c1548e0 con 0x7f129c154500 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.114+0000 7f12a1b9f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f129c15dcf0 0x7f129c06bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.114+0000 7f12a23a0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.114+0000 7f12a1b9f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f129c15dcf0 0x7f129c06bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54304/0 (socket says 172.21.15.129:54304) 2024-03-23T15:47:54.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.114+0000 7f12a1b9f640 1 -- 172.21.15.129:0/1070397354 learned_addr learned my addr 172.21.15.129:0/1070397354 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:54.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.117+0000 7f12a1b9f640 1 -- 172.21.15.129:0/1070397354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 msgr2=0x7f129c15d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:54.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.117+0000 7f12a1b9f640 1 --2- 172.21.15.129:0/1070397354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.117+0000 7f12a1b9f640 1 -- 172.21.15.129:0/1070397354 >> v1:172.21.15.129:6789/0 conn(0x7f129c0ca3c0 legacy=0x7f129c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:54.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.117+0000 7f12a1b9f640 1 -- 172.21.15.129:0/1070397354 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129c0ca900 con 0x7f129c15dcf0 2024-03-23T15:47:54.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.120+0000 7f12a1b9f640 1 --2- 172.21.15.129:0/1070397354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c15dcf0 0x7f129c06bd20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f128c009870 tx=0x7f128c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f925a9a5cab6cdaf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:54.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.124+0000 7f12a139e640 1 -- 172.21.15.129:0/1070397354 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c0317c0 con 0x7f129c15dcf0 2024-03-23T15:47:54.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.124+0000 7f12a139e640 1 -- 172.21.15.129:0/1070397354 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f128c031920 con 0x7f129c15dcf0 2024-03-23T15:47:54.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.124+0000 7f12a139e640 1 -- 172.21.15.129:0/1070397354 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c031cb0 con 0x7f129c15dcf0 2024-03-23T15:47:54.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 -- 172.21.15.129:0/1070397354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c15dcf0 msgr2=0x7f129c06bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:54.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 --2- 172.21.15.129:0/1070397354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c15dcf0 0x7f129c06bd20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f128c009870 tx=0x7f128c02b860 comp rx=0 tx=0).stop 2024-03-23T15:47:54.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 -- 172.21.15.129:0/1070397354 shutdown_connections 2024-03-23T15:47:54.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 --2- 172.21.15.129:0/1070397354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c15dcf0 0x7f129c06bd20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 --2- 172.21.15.129:0/1070397354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 -- 172.21.15.129:0/1070397354 >> 172.21.15.129:0/1070397354 conn(0x7f129c072860 msgr2=0x7f129c072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:54.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 -- 172.21.15.129:0/1070397354 shutdown_connections 2024-03-23T15:47:54.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.125+0000 7f12a33a2640 1 -- 172.21.15.129:0/1070397354 wait complete. 2024-03-23T15:47:54.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.127+0000 7f12a33a2640 1 Processor -- start 2024-03-23T15:47:54.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.127+0000 7f12a33a2640 1 -- start start 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a33a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c0ca3c0 0x7f129c150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a33a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a33a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f129c15dcf0 0x7f129c0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a33a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f129c0d03e0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a33a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f129c150e60 con 0x7f129c15dcf0 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a33a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f129c0d0610 con 0x7f129c154500 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.128+0000 7f12a23a0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c0ca3c0 0x7f129c150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.129+0000 7f12a2ba1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f129c15dcf0 0x7f129c0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.129+0000 7f12a1b9f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.129+0000 7f12a23a0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c0ca3c0 0x7f129c150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54314/0 (socket says 172.21.15.129:54314) 2024-03-23T15:47:54.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.129+0000 7f12a23a0640 1 -- 172.21.15.129:0/3078769463 learned_addr learned my addr 172.21.15.129:0/3078769463 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:54.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.134+0000 7f12a23a0640 1 -- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 msgr2=0x7f129c15c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:54.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.134+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f129c154500 0x7f129c15c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.134+0000 7f12a23a0640 1 -- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f129c15dcf0 msgr2=0x7f129c0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:54.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.134+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f129c15dcf0 0x7f129c0cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.134+0000 7f12a23a0640 1 -- 172.21.15.129:0/3078769463 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128c0094d0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.136+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3078769463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129c0ca3c0 0x7f129c150710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f129800e870 tx=0x7f129800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:54.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.141+0000 7f1292ffd640 1 -- 172.21.15.129:0/3078769463 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12980094f0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.141+0000 7f1292ffd640 1 -- 172.21.15.129:0/3078769463 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f129800c940 con 0x7f129c0ca3c0 2024-03-23T15:47:54.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.141+0000 7f1292ffd640 1 -- 172.21.15.129:0/3078769463 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129800ccf0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.141+0000 7f12a33a2640 1 -- 172.21.15.129:0/3078769463 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129c151160 con 0x7f129c0ca3c0 2024-03-23T15:47:54.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.141+0000 7f12a33a2640 1 -- 172.21.15.129:0/3078769463 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f129c1516e0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.145+0000 7f12a71dc740 1 Processor -- start 2024-03-23T15:47:54.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.146+0000 7f12a71dc740 1 -- start start 2024-03-23T15:47:54.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a71dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f981d70 0x55c58f941b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a71dc740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 0x55c58f982730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a71dc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f982150 0x55c58f981050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a71dc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c58f8af180 con 0x55c58f982150 2024-03-23T15:47:54.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a71dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c58f8ac6b0 con 0x55c58f981d70 2024-03-23T15:47:54.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a71dc740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55c58f8a7e90 con 0x55c58f940810 2024-03-23T15:47:54.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a2ba1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f981d70 0x55c58f941b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.147+0000 7f12a23a0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 0x55c58f982730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.148+0000 7f12a23a0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 0x55c58f982730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35874/0 (socket says 172.21.15.129:35874) 2024-03-23T15:47:54.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.148+0000 7f12a23a0640 1 -- 172.21.15.129:0/282125079 learned_addr learned my addr 172.21.15.129:0/282125079 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:54.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.151+0000 7f12a23a0640 1 -- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f981d70 msgr2=0x55c58f941b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:54.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.151+0000 7f12a23a0640 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f981d70 0x55c58f941b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.151+0000 7f12a23a0640 1 -- 172.21.15.129:0/282125079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f982150 msgr2=0x55c58f981050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:54.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.151+0000 7f12a23a0640 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f982150 0x55c58f981050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.151+0000 7f12a23a0640 1 -- 172.21.15.129:0/282125079 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c58f864d80 con 0x55c58f940810 2024-03-23T15:47:54.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.156+0000 7f12a23a0640 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 0x55c58f982730 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f129800c520 tx=0x7f1298013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891fb94b5eff8a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:54.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.160+0000 7f127ffff640 1 -- 172.21.15.129:0/282125079 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298013470 con 0x55c58f940810 2024-03-23T15:47:54.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.160+0000 7f127ffff640 1 -- 172.21.15.129:0/282125079 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12980135d0 con 0x55c58f940810 2024-03-23T15:47:54.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.160+0000 7f127ffff640 1 -- 172.21.15.129:0/282125079 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298013980 con 0x55c58f940810 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 -- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 msgr2=0x55c58f982730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 0x55c58f982730 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f129800c520 tx=0x7f1298013000 comp rx=0 tx=0).stop 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 -- 172.21.15.129:0/282125079 shutdown_connections 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f982150 0x55c58f981050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f940810 0x55c58f982730 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 --2- 172.21.15.129:0/282125079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f981d70 0x55c58f941b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 -- 172.21.15.129:0/282125079 >> 172.21.15.129:0/282125079 conn(0x55c58f93d980 msgr2=0x55c58f941120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 -- 172.21.15.129:0/282125079 shutdown_connections 2024-03-23T15:47:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.161+0000 7f12a71dc740 1 -- 172.21.15.129:0/282125079 wait complete. 2024-03-23T15:47:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.162+0000 7f12a71dc740 1 -- start start 2024-03-23T15:47:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.162+0000 7f12a71dc740 1 Processor -- start 2024-03-23T15:47:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f981d70 0x55c58f93b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f982150 0x55c58f9952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f967c90 0x55c58f995810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c58f84cd50 con 0x55c58f967c90 2024-03-23T15:47:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c58f57b310 con 0x55c58f982150 2024-03-23T15:47:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55c58f82d8f0 con 0x55c58f981d70 2024-03-23T15:47:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a71dc740 10 client.0 _ll_register_callbacks cb 0x55c58f5b8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T15:47:54.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a2ba1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f981d70 0x55c58f93b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f1292ffd640 1 -- 172.21.15.129:0/3078769463 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1298013740 con 0x7f129c0ca3c0 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a2ba1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f981d70 0x55c58f93b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35882/0 (socket says 172.21.15.129:35882) 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.163+0000 7f12a2ba1640 1 -- 172.21.15.129:0/3567058357 learned_addr learned my addr 172.21.15.129:0/3567058357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.164+0000 7f1292ffd640 1 --2- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f128006b7d0 0x7f128006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.164+0000 7f12a2ba1640 1 --2- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f128006b7d0 0x7f128006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.164+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f982150 0x55c58f9952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:47:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.164+0000 7f12a2ba1640 1 --2- 172.21.15.129:0/3078769463 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f128006b7d0 0x7f128006dc90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1294007090 tx=0x7f1294009c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:54.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.167+0000 7f12a2ba1640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f982150 msgr2=0x55c58f9952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:54.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.167+0000 7f12a2ba1640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c58f982150 0x55c58f9952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.167+0000 7f12a2ba1640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f967c90 msgr2=0x55c58f995810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:47:54.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.167+0000 7f12a2ba1640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c58f967c90 0x55c58f995810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:47:54.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.167+0000 7f12a2ba1640 1 -- 172.21.15.129:0/3567058357 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f129c1516e0 con 0x55c58f981d70 2024-03-23T15:47:54.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.167+0000 7f1292ffd640 1 -- 172.21.15.129:0/3078769463 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f129805b0c0 con 0x7f129c0ca3c0 2024-03-23T15:47:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.169+0000 7f12a2ba1640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c58f981d70 0x55c58f93b650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f129401c910 tx=0x7f129401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=367178944d819392 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:47:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.169+0000 7f127d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.157:3301/0 2024-03-23T15:47:54.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.173+0000 7f127d7fa640 1 -- 172.21.15.129:0/3567058357 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294010040 con 0x55c58f981d70 2024-03-23T15:47:54.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.173+0000 7f127d7fa640 1 -- 172.21.15.129:0/3567058357 <== mon.2 v2:172.21.15.157:3301/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f129401b9e0 con 0x55c58f981d70 2024-03-23T15:47:54.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.173+0000 7f12a71dc740 1 -- 172.21.15.129:0/3567058357 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c58f5a76c0 con 0x55c58f981d70 2024-03-23T15:47:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.175+0000 7f127d7fa640 1 -- 172.21.15.129:0/3567058357 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f129401ce90 con 0x55c58f981d70 2024-03-23T15:47:54.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.175+0000 7f12a71dc740 10 client.4496 fetch_fsmap learned FSMap version 17 2024-03-23T15:47:54.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.175+0000 7f12a71dc740 1 -- 172.21.15.129:0/3567058357 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55c58f8a6510 con 0x55c58f981d70 2024-03-23T15:47:54.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.177+0000 7f127d7fa640 1 -- 172.21.15.129:0/3567058357 <== mon.2 v2:172.21.15.157:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c58f5a76c0 con 0x55c58f981d70 2024-03-23T15:47:54.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f127d7fa640 1 -- 172.21.15.129:0/3567058357 <== mon.2 v2:172.21.15.157:3301/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f129401bba0 con 0x55c58f981d70 2024-03-23T15:47:54.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f12a71dc740 10 client.4496 fetch_fsmap finished waiting for FSMap version 17 2024-03-23T15:47:54.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f12a71dc740 10 client.4496 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T15:47:54.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f12a71dc740 20 client.4496 populate_metadata read hostname 'smithi129' 2024-03-23T15:47:54.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f12a71dc740 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:47:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f12a71dc740 4 client.4496 mds_command: new command op to 4236 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T15:47:54.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:47:54.188+0000 7f12a71dc740 1 -- 172.21.15.129:0/3567058357 --> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c58f5f8270 con 0x55c58f95fd50 2024-03-23T15:48:04.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:04.195+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:48:04.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:04.195+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T15:48:05.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:48:05.826 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:48:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:14.403+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:48:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:14.404+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T15:48:24.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:24.813+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:48:24.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:24.813+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T15:48:34.889 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-03-23T15:48:34.883+0000 139f9640 -1 mds.1.openfiles _load_finish got (2) No such file or directory 2024-03-23T15:48:35.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:35.623+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:48:35.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:35.623+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T15:48:36.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:48:36.013 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:48:47.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:47.225+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:48:47.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:48:47.225+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-23T15:49:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:00.432+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:49:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:00.432+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-23T15:49:06.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:49:06.158 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:49:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:16.848+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:49:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:16.848+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-23T15:49:36.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:49:36.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:39.651+0000 7f12a23a0640 1 -- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 msgr2=0x55c58f992350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.157:6834/568239467, fault. 2024-03-23T15:49:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:39.651+0000 7f12a23a0640 1 --2- 172.21.15.129:0/3567058357 >> [v2:172.21.15.157:6834/568239467,v1:172.21.15.157:6835/568239467] conn(0x55c58f95fd50 0x55c58f992350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-23T15:49:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-23T15:49:55.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:49:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.145+0000 7f2f1db20640 1 Processor -- start 2024-03-23T15:49:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.145+0000 7f2f1db20640 1 -- start start 2024-03-23T15:49:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f1db20640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 0x7f2f1815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f1db20640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2f180c8220 0x7f2f180cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f1db20640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f18143aa0 con 0x7f2f1806f950 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f1db20640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2f1806d2a0 con 0x7f2f180c8220 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f1db20640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1815b1a0 con 0x7f2f181436c0 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f16ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2f180c8220 0x7f2f180cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.146+0000 7f2f177fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 0x7f2f1815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.147+0000 7f2f177fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 0x7f2f1815ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:37664/0 (socket says 172.21.15.129:37664) 2024-03-23T15:49:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.147+0000 7f2f177fe640 1 -- 172.21.15.129:0/3921069673 learned_addr learned my addr 172.21.15.129:0/3921069673 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:49:55.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.149+0000 7f2f177fe640 1 -- 172.21.15.129:0/3921069673 >> v2:172.21.15.157:3300/0 conn(0x7f2f180c8220 msgr2=0x7f2f180cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:49:55.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.149+0000 7f2f177fe640 1 --2- 172.21.15.129:0/3921069673 >> v2:172.21.15.157:3300/0 conn(0x7f2f180c8220 0x7f2f180cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.149+0000 7f2f177fe640 1 -- 172.21.15.129:0/3921069673 >> v1:172.21.15.157:6789/0 conn(0x7f2f1806f950 legacy=0x7f2f1815b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:55.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.149+0000 7f2f177fe640 1 -- 172.21.15.129:0/3921069673 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1806ce10 con 0x7f2f181436c0 2024-03-23T15:49:55.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.152+0000 7f2f177fe640 1 --2- 172.21.15.129:0/3921069673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 0x7f2f1815ac60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009e50 tx=0x7f2f0802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ef5361a6d93a1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:55.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.156+0000 7f2f167fc640 1 -- 172.21.15.129:0/3921069673 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0802cb90 con 0x7f2f181436c0 2024-03-23T15:49:55.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.156+0000 7f2f167fc640 1 -- 172.21.15.129:0/3921069673 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0802ccf0 con 0x7f2f181436c0 2024-03-23T15:49:55.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.156+0000 7f2f167fc640 1 -- 172.21.15.129:0/3921069673 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08034610 con 0x7f2f181436c0 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 -- 172.21.15.129:0/3921069673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 msgr2=0x7f2f1815ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 --2- 172.21.15.129:0/3921069673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 0x7f2f1815ac60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009e50 tx=0x7f2f0802bff0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 -- 172.21.15.129:0/3921069673 shutdown_connections 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 --2- 172.21.15.129:0/3921069673 >> v2:172.21.15.157:3300/0 conn(0x7f2f180c8220 0x7f2f180cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 --2- 172.21.15.129:0/3921069673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f181436c0 0x7f2f1815ac60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 -- 172.21.15.129:0/3921069673 >> 172.21.15.129:0/3921069673 conn(0x7f2f18061b50 msgr2=0x7f2f18061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 -- 172.21.15.129:0/3921069673 shutdown_connections 2024-03-23T15:49:55.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.157+0000 7f2f1db20640 1 -- 172.21.15.129:0/3921069673 wait complete. 2024-03-23T15:49:55.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.158+0000 7f2f1db20640 1 Processor -- start 2024-03-23T15:49:55.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.158+0000 7f2f1db20640 1 -- start start 2024-03-23T15:49:55.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.159+0000 7f2f1db20640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f1806f950 0x7f2f1815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:55.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.159+0000 7f2f1db20640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 0x7f2f181f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:55.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.159+0000 7f2f1db20640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f181436c0 0x7f2f181f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:55.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.159+0000 7f2f1db20640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1815e470 con 0x7f2f181436c0 2024-03-23T15:49:55.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.159+0000 7f2f1db20640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2f181f3ee0 con 0x7f2f180c8220 2024-03-23T15:49:55.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.159+0000 7f2f1db20640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1815dc70 con 0x7f2f1806f950 2024-03-23T15:49:55.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.160+0000 7f2f16ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 0x7f2f181f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:55.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.160+0000 7f2f177fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f1806f950 0x7f2f1815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:55.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.160+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f181436c0 0x7f2f181f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:55.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.160+0000 7f2f16ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 0x7f2f181f3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:34492/0 (socket says 172.21.15.129:34492) 2024-03-23T15:49:55.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.160+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f181436c0 0x7f2f181f39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49276/0 (socket says 172.21.15.129:49276) 2024-03-23T15:49:55.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.160+0000 7f2f16ffd640 1 -- 172.21.15.129:0/2129532583 learned_addr learned my addr 172.21.15.129:0/2129532583 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:49:55.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.165+0000 7f2f16ffd640 1 -- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f1806f950 msgr2=0x7f2f1815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:55.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.165+0000 7f2f16ffd640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f1806f950 0x7f2f1815c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.165+0000 7f2f16ffd640 1 -- 172.21.15.129:0/2129532583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f181436c0 msgr2=0x7f2f181f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:55.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.165+0000 7f2f16ffd640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f181436c0 0x7f2f181f39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.165+0000 7f2f16ffd640 1 -- 172.21.15.129:0/2129532583 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f08009b30 con 0x7f2f180c8220 2024-03-23T15:49:55.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.170+0000 7f2f16ffd640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 0x7f2f181f3460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000e9e0 tx=0x7f2f0000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:55.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.176+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f00009670 con 0x7f2f180c8220 2024-03-23T15:49:55.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.176+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0000c970 con 0x7f2f180c8220 2024-03-23T15:49:55.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.176+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000cd20 con 0x7f2f180c8220 2024-03-23T15:49:55.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.176+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1815c2c0 con 0x7f2f180c8220 2024-03-23T15:49:55.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.176+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1815deb0 con 0x7f2f180c8220 2024-03-23T15:49:55.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.177+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1815c450 con 0x7f2f180c8220 2024-03-23T15:49:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.211+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f00013470 con 0x7f2f180c8220 2024-03-23T15:49:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.211+0000 7f2f0ffff640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2ef806b890 0x7f2ef806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.211+0000 7f2f177fe640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2ef806b890 0x7f2ef806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.212+0000 7f2f177fe640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2ef806b890 0x7f2ef806dd50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f08035000 tx=0x7f2f08030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:55.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.457+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2f0000ce80 con 0x7f2f180c8220 2024-03-23T15:49:55.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.474+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f000555c0 con 0x7f2f180c8220 2024-03-23T15:49:55.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.715+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f18075740 con 0x7f2f180c8220 2024-03-23T15:49:55.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.752+0000 7f2f0ffff640 1 -- 172.21.15.129:0/2129532583 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2f00021090 con 0x7f2f180c8220 2024-03-23T15:49:55.755 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:49:55.755 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:49:55.756 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:49:55.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2ef806b890 msgr2=0x7f2ef806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:55.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2ef806b890 0x7f2ef806dd50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2f08035000 tx=0x7f2f08030000 comp rx=0 tx=0).stop 2024-03-23T15:49:55.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 msgr2=0x7f2f181f3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 0x7f2f181f3460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000e9e0 tx=0x7f2f0000eea0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 shutdown_connections 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f181436c0 0x7f2f181f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2ef806b890 0x7f2ef806dd50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f180c8220 0x7f2f181f3460 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 --2- 172.21.15.129:0/2129532583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f1806f950 0x7f2f1815c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 >> 172.21.15.129:0/2129532583 conn(0x7f2f18061b50 msgr2=0x7f2f180ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 shutdown_connections 2024-03-23T15:49:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:55.754+0000 7f2f1db20640 1 -- 172.21.15.129:0/2129532583 wait complete. 2024-03-23T15:49:56.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:49:56.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:49:57.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:49:57.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.477+0000 7fdf68b5e640 1 Processor -- start 2024-03-23T15:49:57.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.477+0000 7fdf68b5e640 1 -- start start 2024-03-23T15:49:57.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.478+0000 7fdf68b5e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdf6415e010 0x7fdf64165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:57.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.478+0000 7fdf68b5e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6406c830 con 0x7fdf6406c410 2024-03-23T15:49:57.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.478+0000 7fdf68b5e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf640cc370 con 0x7fdf6406c030 2024-03-23T15:49:57.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.478+0000 7fdf68b5e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdf6406cae0 con 0x7fdf6415e010 2024-03-23T15:49:57.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.479+0000 7fdf61d9b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fdf6406c410 0x7fdf6415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38574/0 (socket says 172.21.15.129:38574) 2024-03-23T15:49:57.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.479+0000 7fdf61d9b640 1 -- 172.21.15.129:0/3601261357 learned_addr learned my addr 172.21.15.129:0/3601261357 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:49:57.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.479+0000 7fdf62d9d640 1 --2- 172.21.15.129:0/3601261357 >> v2:172.21.15.157:3300/0 conn(0x7fdf6415e010 0x7fdf64165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:57.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.481+0000 7fdf6159a640 1 -- 172.21.15.129:0/3601261357 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2784612209 0 0) 0x7fdf6406c830 con 0x7fdf6406c410 2024-03-23T15:49:57.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.481+0000 7fdf6159a640 1 -- 172.21.15.129:0/3601261357 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf4c003520 con 0x7fdf6406c410 2024-03-23T15:49:57.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.484+0000 7fdf62d9d640 1 -- 172.21.15.129:0/3601261357 >> v1:172.21.15.157:6789/0 conn(0x7fdf6406c030 legacy=0x7fdf640cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:57.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.484+0000 7fdf62d9d640 1 -- 172.21.15.129:0/3601261357 >> v1:172.21.15.129:6789/0 conn(0x7fdf6406c410 legacy=0x7fdf6415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:57.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.484+0000 7fdf62d9d640 1 -- 172.21.15.129:0/3601261357 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf640cabc0 con 0x7fdf6415e010 2024-03-23T15:49:57.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.487+0000 7fdf62d9d640 1 --2- 172.21.15.129:0/3601261357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf64165c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009d90 tx=0x7fdf5c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9978860109946e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:57.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.492+0000 7fdf6159a640 1 -- 172.21.15.129:0/3601261357 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c02cd00 con 0x7fdf6415e010 2024-03-23T15:49:57.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.492+0000 7fdf6159a640 1 -- 172.21.15.129:0/3601261357 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c02ce60 con 0x7fdf6415e010 2024-03-23T15:49:57.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.492+0000 7fdf6159a640 1 -- 172.21.15.129:0/3601261357 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c034660 con 0x7fdf6415e010 2024-03-23T15:49:57.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 -- 172.21.15.129:0/3601261357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 msgr2=0x7fdf64165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:57.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/3601261357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf64165c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009d90 tx=0x7fdf5c02bf50 comp rx=0 tx=0).stop 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 -- 172.21.15.129:0/3601261357 shutdown_connections 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/3601261357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf64165c60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 -- 172.21.15.129:0/3601261357 >> 172.21.15.129:0/3601261357 conn(0x7fdf640cfb20 msgr2=0x7fdf640cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 -- 172.21.15.129:0/3601261357 shutdown_connections 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.493+0000 7fdf68b5e640 1 -- 172.21.15.129:0/3601261357 wait complete. 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 Processor -- start 2024-03-23T15:49:57.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 -- start start 2024-03-23T15:49:57.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdf6406c030 0x7fdf641576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:57.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 0x7fdf641f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:57.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf6415a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:57.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf641f42a0 con 0x7fdf6406c410 2024-03-23T15:49:57.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdf641efde0 con 0x7fdf6415e010 2024-03-23T15:49:57.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.495+0000 7fdf68b5e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdf641f0d10 con 0x7fdf6406c030 2024-03-23T15:49:57.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.496+0000 7fdf61d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 0x7fdf641f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:57.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.496+0000 7fdf6259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdf6406c030 0x7fdf641576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:57.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.496+0000 7fdf61d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 0x7fdf641f07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49280/0 (socket says 172.21.15.129:49280) 2024-03-23T15:49:57.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.496+0000 7fdf61d9b640 1 -- 172.21.15.129:0/517370199 learned_addr learned my addr 172.21.15.129:0/517370199 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:49:57.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.496+0000 7fdf62d9d640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf6415a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:57.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.500+0000 7fdf61d9b640 1 -- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdf6406c030 msgr2=0x7fdf641576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:57.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.500+0000 7fdf61d9b640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdf6406c030 0x7fdf641576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:57.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.500+0000 7fdf61d9b640 1 -- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 msgr2=0x7fdf6415a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:57.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.500+0000 7fdf61d9b640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf6415a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:57.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.500+0000 7fdf61d9b640 1 -- 172.21.15.129:0/517370199 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c009a70 con 0x7fdf6406c410 2024-03-23T15:49:57.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.502+0000 7fdf61d9b640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 0x7fdf641f07d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c500 tx=0x7fdf5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:57.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.504+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58005ce0 con 0x7fdf6406c410 2024-03-23T15:49:57.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.504+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf64154f30 con 0x7fdf6406c410 2024-03-23T15:49:57.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.504+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf641effd0 con 0x7fdf6406c410 2024-03-23T15:49:57.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.505+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf640686f0 con 0x7fdf6406c410 2024-03-23T15:49:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.762+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf58005e40 con 0x7fdf6406c410 2024-03-23T15:49:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.762+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf580072f0 con 0x7fdf6406c410 2024-03-23T15:49:57.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.763+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdf58007510 con 0x7fdf6406c410 2024-03-23T15:49:57.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.763+0000 7fdf52ffd640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf4406b810 0x7fdf4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:57.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.764+0000 7fdf6259c640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf4406b810 0x7fdf4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:57.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.764+0000 7fdf6259c640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf4406b810 0x7fdf4406dcd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdf540016e0 tx=0x7fdf54009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:57.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.779+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdf580588f0 con 0x7fdf6406c410 2024-03-23T15:49:57.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:57.779+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdf58054b90 con 0x7fdf6406c410 2024-03-23T15:49:58.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.001+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf64155450 con 0x7fdf6406c410 2024-03-23T15:49:58.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.024+0000 7fdf52ffd640 1 -- 172.21.15.129:0/517370199 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fdf58021090 con 0x7fdf6406c410 2024-03-23T15:49:58.027 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:49:58.027 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:49:58.027 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:49:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf4406b810 msgr2=0x7fdf4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf4406b810 0x7fdf4406dcd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdf540016e0 tx=0x7fdf54009040 comp rx=0 tx=0).stop 2024-03-23T15:49:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 msgr2=0x7fdf641f07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 0x7fdf641f07d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c500 tx=0x7fdf5800c9c0 comp rx=0 tx=0).stop 2024-03-23T15:49:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 shutdown_connections 2024-03-23T15:49:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf4406b810 0x7fdf4406dcd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:58.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdf6415e010 0x7fdf6415a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:58.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6406c410 0x7fdf641f07d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:58.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 --2- 172.21.15.129:0/517370199 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdf6406c030 0x7fdf641576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:58.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 >> 172.21.15.129:0/517370199 conn(0x7fdf640cfb20 msgr2=0x7fdf640cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:49:58.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 shutdown_connections 2024-03-23T15:49:58.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:58.027+0000 7fdf68b5e640 1 -- 172.21.15.129:0/517370199 wait complete. 2024-03-23T15:49:58.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:49:58.649 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:49:59.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:49:59.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.752+0000 7fc953eda640 1 Processor -- start 2024-03-23T15:49:59.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.752+0000 7fc953eda640 1 -- start start 2024-03-23T15:49:59.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.753+0000 7fc953eda640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc94c15dc60 0x7fc94c06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:59.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.753+0000 7fc953eda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c06f570 0x7fc94c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:59.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.753+0000 7fc953eda640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc94c06d2c0 con 0x7fc94c15e040 2024-03-23T15:49:59.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.753+0000 7fc953eda640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc94c15e740 con 0x7fc94c15dc60 2024-03-23T15:49:59.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.753+0000 7fc953eda640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c0ccd00 con 0x7fc94c06f570 2024-03-23T15:49:59.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.754+0000 7fc951c76640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc94c15dc60 0x7fc94c06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:59.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.754+0000 7fc951475640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc94c15e040 0x7fc94c0cdda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38588/0 (socket says 172.21.15.129:38588) 2024-03-23T15:49:59.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.754+0000 7fc951475640 1 -- 172.21.15.129:0/1934861214 learned_addr learned my addr 172.21.15.129:0/1934861214 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:49:59.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.754+0000 7fc952477640 1 --2- 172.21.15.129:0/1934861214 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c06f570 0x7fc94c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:59.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.757+0000 7fc952477640 1 -- 172.21.15.129:0/1934861214 >> v2:172.21.15.157:3300/0 conn(0x7fc94c15dc60 msgr2=0x7fc94c06cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:49:59.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.757+0000 7fc952477640 1 --2- 172.21.15.129:0/1934861214 >> v2:172.21.15.157:3300/0 conn(0x7fc94c15dc60 0x7fc94c06cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:59.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.757+0000 7fc952477640 1 -- 172.21.15.129:0/1934861214 >> v1:172.21.15.129:6789/0 conn(0x7fc94c15e040 legacy=0x7fc94c0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:59.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.757+0000 7fc952477640 1 -- 172.21.15.129:0/1934861214 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94c0ccfe0 con 0x7fc94c06f570 2024-03-23T15:49:59.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.759+0000 7fc952477640 1 --2- 172.21.15.129:0/1934861214 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c06f570 0x7fc94c0cb7c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc944009d90 tx=0x7fc94402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=417ddf7b57bfd454 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:59.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.763+0000 7fc950c74640 1 -- 172.21.15.129:0/1934861214 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94402cbc0 con 0x7fc94c06f570 2024-03-23T15:49:59.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.763+0000 7fc950c74640 1 -- 172.21.15.129:0/1934861214 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc94402cd20 con 0x7fc94c06f570 2024-03-23T15:49:59.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.763+0000 7fc950c74640 1 -- 172.21.15.129:0/1934861214 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944034660 con 0x7fc94c06f570 2024-03-23T15:49:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 -- 172.21.15.129:0/1934861214 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c06f570 msgr2=0x7fc94c0cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 --2- 172.21.15.129:0/1934861214 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c06f570 0x7fc94c0cb7c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc944009d90 tx=0x7fc94402be10 comp rx=0 tx=0).stop 2024-03-23T15:49:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 -- 172.21.15.129:0/1934861214 shutdown_connections 2024-03-23T15:49:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 --2- 172.21.15.129:0/1934861214 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c06f570 0x7fc94c0cb7c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 --2- 172.21.15.129:0/1934861214 >> v2:172.21.15.157:3300/0 conn(0x7fc94c15dc60 0x7fc94c06cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 -- 172.21.15.129:0/1934861214 >> 172.21.15.129:0/1934861214 conn(0x7fc94c071300 msgr2=0x7fc94c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:49:59.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 -- 172.21.15.129:0/1934861214 shutdown_connections 2024-03-23T15:49:59.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.764+0000 7fc953eda640 1 -- 172.21.15.129:0/1934861214 wait complete. 2024-03-23T15:49:59.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.765+0000 7fc953eda640 1 Processor -- start 2024-03-23T15:49:59.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.765+0000 7fc953eda640 1 -- start start 2024-03-23T15:49:59.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc953eda640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc94c06f570 0x7fc94c1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:59.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc953eda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c15dc60 0x7fc94c1f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:59.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc953eda640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 0x7fc94c1e3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:59.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc953eda640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c0ae170 con 0x7fc94c15e040 2024-03-23T15:49:59.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc953eda640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc94c0c5f20 con 0x7fc94c06f570 2024-03-23T15:49:59.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc953eda640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc94c0c6200 con 0x7fc94c15dc60 2024-03-23T15:49:59.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.766+0000 7fc951c76640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc94c06f570 0x7fc94c1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:59.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.767+0000 7fc952477640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 0x7fc94c1e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:59.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.767+0000 7fc951475640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c15dc60 0x7fc94c1f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:59.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.767+0000 7fc952477640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 0x7fc94c1e3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49288/0 (socket says 172.21.15.129:49288) 2024-03-23T15:49:59.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.767+0000 7fc952477640 1 -- 172.21.15.129:0/3641025434 learned_addr learned my addr 172.21.15.129:0/3641025434 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:49:59.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.770+0000 7fc952477640 1 -- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c15dc60 msgr2=0x7fc94c1f12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:59.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.770+0000 7fc952477640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c15dc60 0x7fc94c1f12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:59.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.770+0000 7fc952477640 1 -- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc94c06f570 msgr2=0x7fc94c1efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:49:59.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.770+0000 7fc952477640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc94c06f570 0x7fc94c1efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:49:59.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.770+0000 7fc952477640 1 -- 172.21.15.129:0/3641025434 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc944009a70 con 0x7fc94c15e040 2024-03-23T15:49:59.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.773+0000 7fc952477640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 0x7fc94c1e3890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc944009eb0 tx=0x7fc944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:49:59.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.777+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94403e610 con 0x7fc94c15e040 2024-03-23T15:49:59.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.777+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc94402ce90 con 0x7fc94c15e040 2024-03-23T15:49:59.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.777+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94c0c64e0 con 0x7fc94c15e040 2024-03-23T15:49:59.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.777+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94403e9b0 con 0x7fc94c15e040 2024-03-23T15:49:59.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.777+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94c0c6aa0 con 0x7fc94c15e040 2024-03-23T15:49:59.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.778+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc94c06bbc0 con 0x7fc94c15e040 2024-03-23T15:49:59.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.801+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc94403b050 con 0x7fc94c15e040 2024-03-23T15:49:59.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.802+0000 7fc9427fc640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc93006b890 0x7fc93006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:49:59.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.802+0000 7fc951c76640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc93006b890 0x7fc93006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:49:59.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:49:59.803+0000 7fc951c76640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc93006b890 0x7fc93006dd50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc948009aa0 tx=0x7fc948009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:00.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.071+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc94407b4f0 con 0x7fc94c15e040 2024-03-23T15:50:00.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.072+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc944077790 con 0x7fc94c15e040 2024-03-23T15:50:00.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.307+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc94c0cc5e0 con 0x7fc94c15e040 2024-03-23T15:50:00.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.320+0000 7fc9427fc640 1 -- 172.21.15.129:0/3641025434 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc94404e090 con 0x7fc94c15e040 2024-03-23T15:50:00.323 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:00.323 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:00.323 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:00.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc93006b890 msgr2=0x7fc93006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc93006b890 0x7fc93006dd50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc948009aa0 tx=0x7fc948009190 comp rx=0 tx=0).stop 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 msgr2=0x7fc94c1e3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 0x7fc94c1e3890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc944009eb0 tx=0x7fc944004a40 comp rx=0 tx=0).stop 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 shutdown_connections 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc94c15e040 0x7fc94c1e3890 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc94c15dc60 0x7fc94c1f12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc93006b890 0x7fc93006dd50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 --2- 172.21.15.129:0/3641025434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc94c06f570 0x7fc94c1efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 >> 172.21.15.129:0/3641025434 conn(0x7fc94c071300 msgr2=0x7fc94c0ca4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 shutdown_connections 2024-03-23T15:50:00.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:00.322+0000 7fc953eda640 1 -- 172.21.15.129:0/3641025434 wait complete. 2024-03-23T15:50:00.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:00.940 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:01.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.046+0000 7f230ef3a640 1 Processor -- start 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.046+0000 7f230ef3a640 1 -- start start 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.047+0000 7f230ef3a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f23081544b0 0x7f230815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.047+0000 7f230ef3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f230806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.047+0000 7f230ef3a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23080cbcb0 con 0x7f230806c0b0 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.047+0000 7f230ef3a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f23080c5be0 con 0x7f23081544b0 2024-03-23T15:50:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.047+0000 7f230ef3a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f230815b280 con 0x7f23080caa00 2024-03-23T15:50:02.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.048+0000 7f2307fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f230806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:02.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.048+0000 7f230d4d7640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f230806c0b0 0x7f23080c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:39048/0 (socket says 172.21.15.129:39048) 2024-03-23T15:50:02.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.048+0000 7f230d4d7640 1 -- 172.21.15.129:0/2069236226 learned_addr learned my addr 172.21.15.129:0/2069236226 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:02.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.048+0000 7f230ccd6640 1 --2- 172.21.15.129:0/2069236226 >> v2:172.21.15.157:3300/0 conn(0x7f23081544b0 0x7f230815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.051+0000 7f2307fff640 1 -- 172.21.15.129:0/2069236226 >> v2:172.21.15.157:3300/0 conn(0x7f23081544b0 msgr2=0x7f230815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:02.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.051+0000 7f2307fff640 1 --2- 172.21.15.129:0/2069236226 >> v2:172.21.15.157:3300/0 conn(0x7f23081544b0 0x7f230815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.051+0000 7f2307fff640 1 -- 172.21.15.129:0/2069236226 >> v1:172.21.15.129:6789/0 conn(0x7f230806c0b0 legacy=0x7f23080c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:02.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.051+0000 7f2307fff640 1 -- 172.21.15.129:0/2069236226 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230815a9e0 con 0x7f23080caa00 2024-03-23T15:50:02.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.053+0000 7f2307fff640 1 --2- 172.21.15.129:0/2069236226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f230806bb70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f22f40097a0 tx=0x7f22f402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480b96aec1762a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.056+0000 7f23077fe640 1 -- 172.21.15.129:0/2069236226 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4030610 con 0x7f23080caa00 2024-03-23T15:50:02.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.058+0000 7f23077fe640 1 -- 172.21.15.129:0/2069236226 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f4030770 con 0x7f23080caa00 2024-03-23T15:50:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.058+0000 7f23077fe640 1 -- 172.21.15.129:0/2069236226 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4030b00 con 0x7f23080caa00 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 -- 172.21.15.129:0/2069236226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 msgr2=0x7f230806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2069236226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f230806bb70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f22f40097a0 tx=0x7f22f402b960 comp rx=0 tx=0).stop 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 -- 172.21.15.129:0/2069236226 shutdown_connections 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2069236226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f230806bb70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2069236226 >> v2:172.21.15.157:3300/0 conn(0x7f23081544b0 0x7f230815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 -- 172.21.15.129:0/2069236226 >> 172.21.15.129:0/2069236226 conn(0x7f2308061b50 msgr2=0x7f2308061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 -- 172.21.15.129:0/2069236226 shutdown_connections 2024-03-23T15:50:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.059+0000 7f230ef3a640 1 -- 172.21.15.129:0/2069236226 wait complete. 2024-03-23T15:50:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.061+0000 7f230ef3a640 1 Processor -- start 2024-03-23T15:50:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.061+0000 7f230ef3a640 1 -- start start 2024-03-23T15:50:02.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.062+0000 7f230ef3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f230806c0b0 0x7f23081561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:02.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.062+0000 7f230ef3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f23080cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:02.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.062+0000 7f230ef3a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23081544b0 0x7f2308141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:02.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.062+0000 7f230ef3a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23081efd00 con 0x7f230806c0b0 2024-03-23T15:50:02.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.062+0000 7f230ef3a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f23080cfea0 con 0x7f23081544b0 2024-03-23T15:50:02.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.062+0000 7f230ef3a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f23080d0180 con 0x7f23080caa00 2024-03-23T15:50:02.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.063+0000 7f2307fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f23080cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:02.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.064+0000 7f2307fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f23080cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:42518/0 (socket says 172.21.15.129:42518) 2024-03-23T15:50:02.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.064+0000 7f2307fff640 1 -- 172.21.15.129:0/2002970985 learned_addr learned my addr 172.21.15.129:0/2002970985 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.065+0000 7f230d4d7640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23081544b0 0x7f2308141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.065+0000 7f230ccd6640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f230806c0b0 0x7f23081561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:02.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.068+0000 7f2307fff640 1 -- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23081544b0 msgr2=0x7f2308141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:02.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.068+0000 7f2307fff640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23081544b0 0x7f2308141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.068+0000 7f2307fff640 1 -- 172.21.15.129:0/2002970985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f230806c0b0 msgr2=0x7f23081561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.068+0000 7f2307fff640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f230806c0b0 0x7f23081561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.068+0000 7f2307fff640 1 -- 172.21.15.129:0/2002970985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f4009400 con 0x7f23080caa00 2024-03-23T15:50:02.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.070+0000 7f2307fff640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f23080cdcb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f22f40098c0 tx=0x7f22f4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:02.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.075+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f403e040 con 0x7f23080caa00 2024-03-23T15:50:02.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.075+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f4039cb0 con 0x7f23080caa00 2024-03-23T15:50:02.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.075+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23081effc0 con 0x7f23080caa00 2024-03-23T15:50:02.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.075+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4044630 con 0x7f23080caa00 2024-03-23T15:50:02.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.076+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23081f0580 con 0x7f23080caa00 2024-03-23T15:50:02.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.077+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2308075740 con 0x7f23080caa00 2024-03-23T15:50:02.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.357+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22f4043390 con 0x7f23080caa00 2024-03-23T15:50:02.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.357+0000 7f23057fa640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f22ec06b840 0x7f22ec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:02.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.357+0000 7f230ccd6640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f22ec06b840 0x7f22ec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:02.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.358+0000 7f230ccd6640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f22ec06b840 0x7f22ec06dd00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009aa0 tx=0x7f22fc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:02.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.362+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f22f4039e20 con 0x7f23080caa00 2024-03-23T15:50:02.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.380+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22f4077490 con 0x7f23080caa00 2024-03-23T15:50:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.604+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23080cb540 con 0x7f23080caa00 2024-03-23T15:50:02.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.663+0000 7f23057fa640 1 -- 172.21.15.129:0/2002970985 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f22f4049020 con 0x7f23080caa00 2024-03-23T15:50:02.667 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:02.667 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:02.668 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:02.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f22ec06b840 msgr2=0x7f22ec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:02.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f22ec06b840 0x7f22ec06dd00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009aa0 tx=0x7f22fc009190 comp rx=0 tx=0).stop 2024-03-23T15:50:02.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 msgr2=0x7f23080cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:02.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f23080cdcb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f22f40098c0 tx=0x7f22f4004730 comp rx=0 tx=0).stop 2024-03-23T15:50:02.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 shutdown_connections 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f22ec06b840 0x7f22ec06dd00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23081544b0 0x7f2308141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23080caa00 0x7f23080cdcb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 --2- 172.21.15.129:0/2002970985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f230806c0b0 0x7f23081561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 >> 172.21.15.129:0/2002970985 conn(0x7f2308061b50 msgr2=0x7f23080b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 shutdown_connections 2024-03-23T15:50:02.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:02.666+0000 7f230ef3a640 1 -- 172.21.15.129:0/2002970985 wait complete. 2024-03-23T15:50:03.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:03.286 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:04.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:04.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.386+0000 7f621109a640 1 Processor -- start 2024-03-23T15:50:04.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.386+0000 7f621109a640 1 -- start start 2024-03-23T15:50:04.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.387+0000 7f621109a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f620c1544b0 0x7f620c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:04.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.387+0000 7f621109a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f620c0c6000 0x7f620c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.387+0000 7f621109a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f620c0cbcb0 con 0x7f620c0caa00 2024-03-23T15:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.387+0000 7f621109a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f620c0c5be0 con 0x7f620c0c6000 2024-03-23T15:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.387+0000 7f621109a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f620c0c69b0 con 0x7f620c1544b0 2024-03-23T15:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.388+0000 7f620b59e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f620c0c6000 0x7f620c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.388+0000 7f620ad9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f620c1544b0 0x7f620c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.388+0000 7f620b59e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f620c0c6000 0x7f620c0c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54086/0 (socket says 172.21.15.129:54086) 2024-03-23T15:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.388+0000 7f620b59e640 1 -- 172.21.15.129:0/3636931574 learned_addr learned my addr 172.21.15.129:0/3636931574 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:04.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.391+0000 7f620ad9d640 1 -- 172.21.15.129:0/3636931574 >> v2:172.21.15.129:3300/0 conn(0x7f620c0c6000 msgr2=0x7f620c0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.391+0000 7f620ad9d640 1 --2- 172.21.15.129:0/3636931574 >> v2:172.21.15.129:3300/0 conn(0x7f620c0c6000 0x7f620c0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.391+0000 7f620ad9d640 1 -- 172.21.15.129:0/3636931574 >> v1:172.21.15.129:6789/0 conn(0x7f620c0caa00 legacy=0x7f620c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.391+0000 7f620ad9d640 1 -- 172.21.15.129:0/3636931574 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c15b280 con 0x7f620c1544b0 2024-03-23T15:50:04.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.393+0000 7f620ad9d640 1 --2- 172.21.15.129:0/3636931574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c1544b0 0x7f620c15a2e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f620c06cd70 tx=0x7f61fc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38ac24ab78871981 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:04.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.397+0000 7f6209d9b640 1 -- 172.21.15.129:0/3636931574 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc030610 con 0x7f620c1544b0 2024-03-23T15:50:04.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.397+0000 7f6209d9b640 1 -- 172.21.15.129:0/3636931574 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc030770 con 0x7f620c1544b0 2024-03-23T15:50:04.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.397+0000 7f6209d9b640 1 -- 172.21.15.129:0/3636931574 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc030b00 con 0x7f620c1544b0 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 -- 172.21.15.129:0/3636931574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c1544b0 msgr2=0x7f620c15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 --2- 172.21.15.129:0/3636931574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c1544b0 0x7f620c15a2e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f620c06cd70 tx=0x7f61fc02b960 comp rx=0 tx=0).stop 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 -- 172.21.15.129:0/3636931574 shutdown_connections 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 --2- 172.21.15.129:0/3636931574 >> v2:172.21.15.129:3300/0 conn(0x7f620c0c6000 0x7f620c0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 --2- 172.21.15.129:0/3636931574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c1544b0 0x7f620c15a2e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 -- 172.21.15.129:0/3636931574 >> 172.21.15.129:0/3636931574 conn(0x7f620c061b50 msgr2=0x7f620c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 -- 172.21.15.129:0/3636931574 shutdown_connections 2024-03-23T15:50:04.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.398+0000 7f621109a640 1 -- 172.21.15.129:0/3636931574 wait complete. 2024-03-23T15:50:04.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.399+0000 7f621109a640 1 Processor -- start 2024-03-23T15:50:04.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.399+0000 7f621109a640 1 -- start start 2024-03-23T15:50:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.400+0000 7f621109a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0c6000 0x7f620c0cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.400+0000 7f621109a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f620c0caa00 0x7f620c154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.400+0000 7f621109a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f620c1544b0 0x7f620c0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.400+0000 7f621109a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f620c1574e0 con 0x7f620c0caa00 2024-03-23T15:50:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.400+0000 7f621109a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f620c0d0390 con 0x7f620c0c6000 2024-03-23T15:50:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.400+0000 7f621109a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f620c1464d0 con 0x7f620c1544b0 2024-03-23T15:50:04.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.401+0000 7f620a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f620c0caa00 0x7f620c154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:04.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.401+0000 7f620ad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0c6000 0x7f620c0cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:04.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.401+0000 7f620b59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f620c1544b0 0x7f620c0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:04.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.401+0000 7f620a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f620c0caa00 0x7f620c154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54096/0 (socket says 172.21.15.129:54096) 2024-03-23T15:50:04.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.401+0000 7f620a59c640 1 -- 172.21.15.129:0/2710873170 learned_addr learned my addr 172.21.15.129:0/2710873170 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:04.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.405+0000 7f620ad9d640 1 -- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f620c1544b0 msgr2=0x7f620c0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:04.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.405+0000 7f620ad9d640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f620c1544b0 0x7f620c0cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:04.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.405+0000 7f620ad9d640 1 -- 172.21.15.129:0/2710873170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f620c0caa00 msgr2=0x7f620c154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:04.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.405+0000 7f620ad9d640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f620c0caa00 0x7f620c154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:04.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.405+0000 7f620ad9d640 1 -- 172.21.15.129:0/2710873170 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc009400 con 0x7f620c0c6000 2024-03-23T15:50:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.409+0000 7f620ad9d640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0c6000 0x7f620c0cfe50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f61fc033000 tx=0x7f61fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:04.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.413+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc03e070 con 0x7f620c0c6000 2024-03-23T15:50:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.413+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc030e30 con 0x7f620c0c6000 2024-03-23T15:50:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.414+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620c1460b0 con 0x7f620c0c6000 2024-03-23T15:50:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.414+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc039c70 con 0x7f620c0c6000 2024-03-23T15:50:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.414+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620c156860 con 0x7f620c0c6000 2024-03-23T15:50:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.414+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620c157620 con 0x7f620c0c6000 2024-03-23T15:50:04.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.438+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f61fc0423f0 con 0x7f620c0c6000 2024-03-23T15:50:04.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.438+0000 7f61f37fe640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f61f806b860 0x7f61f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:04.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.438+0000 7f620a59c640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f61f806b860 0x7f61f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:04.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.439+0000 7f620a59c640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f61f806b860 0x7f61f806dd20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f620c0ca1c0 tx=0x7f6200009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:04.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.441+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f61fc07aba0 con 0x7f620c0c6000 2024-03-23T15:50:04.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.697+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f61fc076e40 con 0x7f620c0c6000 2024-03-23T15:50:04.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:04.915+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f620c075740 con 0x7f620c0c6000 2024-03-23T15:50:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.004+0000 7f61f37fe640 1 -- 172.21.15.129:0/2710873170 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f61fc048090 con 0x7f620c0c6000 2024-03-23T15:50:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:05.007 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:05.008 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f61f806b860 msgr2=0x7f61f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f61f806b860 0x7f61f806dd20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f620c0ca1c0 tx=0x7f6200009040 comp rx=0 tx=0).stop 2024-03-23T15:50:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0c6000 msgr2=0x7f620c0cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0c6000 0x7f620c0cfe50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f61fc033000 tx=0x7f61fc0046c0 comp rx=0 tx=0).stop 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 shutdown_connections 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f620c1544b0 0x7f620c0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f620c0caa00 0x7f620c154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f61f806b860 0x7f61f806dd20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 --2- 172.21.15.129:0/2710873170 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0c6000 0x7f620c0cfe50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 >> 172.21.15.129:0/2710873170 conn(0x7f620c061b50 msgr2=0x7f620c0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 shutdown_connections 2024-03-23T15:50:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:05.006+0000 7f621109a640 1 -- 172.21.15.129:0/2710873170 wait complete. 2024-03-23T15:50:05.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:05.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:06.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:50:06.359 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:50:06.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:06.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.760+0000 7f3f9b42e640 1 Processor -- start 2024-03-23T15:50:06.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.761+0000 7f3f9b42e640 1 -- start start 2024-03-23T15:50:06.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.761+0000 7f3f9b42e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3f9415a2e0 0x7f3f9406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.761+0000 7f3f9b42e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.761+0000 7f3f9b42e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f940cbcb0 con 0x7f3f941544b0 2024-03-23T15:50:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.761+0000 7f3f9b42e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3f940c5be0 con 0x7f3f9415a2e0 2024-03-23T15:50:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.762+0000 7f3f9b42e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f9415b280 con 0x7f3f9406c0b0 2024-03-23T15:50:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.762+0000 7f3f989c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3f9415a2e0 0x7f3f9406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.762+0000 7f3f999cb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:06.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.762+0000 7f3f991ca640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3f941544b0 0x7f3f940caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:34628/0 (socket says 172.21.15.129:34628) 2024-03-23T15:50:06.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.762+0000 7f3f991ca640 1 -- 172.21.15.129:0/1778736703 learned_addr learned my addr 172.21.15.129:0/1778736703 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:06.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.765+0000 7f3f8bfff640 1 -- 172.21.15.129:0/1778736703 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480744627 0 0) 0x7f3f940cbcb0 con 0x7f3f941544b0 2024-03-23T15:50:06.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.766+0000 7f3f8bfff640 1 -- 172.21.15.129:0/1778736703 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f80003520 con 0x7f3f941544b0 2024-03-23T15:50:06.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.766+0000 7f3f989c9640 1 -- 172.21.15.129:0/1778736703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 msgr2=0x7f3f940c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:06.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.766+0000 7f3f989c9640 1 --2- 172.21.15.129:0/1778736703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:06.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.766+0000 7f3f989c9640 1 -- 172.21.15.129:0/1778736703 >> v1:172.21.15.157:6789/0 conn(0x7f3f941544b0 legacy=0x7f3f940caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:06.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.766+0000 7f3f989c9640 1 -- 172.21.15.129:0/1778736703 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9415a9e0 con 0x7f3f9415a2e0 2024-03-23T15:50:06.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.769+0000 7f3f989c9640 1 --2- 172.21.15.129:0/1778736703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f9406bb70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3f840097a0 tx=0x7f3f8402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4bb073892ccc9f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:06.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.774+0000 7f3f8bfff640 1 -- 172.21.15.129:0/1778736703 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84030610 con 0x7f3f9415a2e0 2024-03-23T15:50:06.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.774+0000 7f3f8bfff640 1 -- 172.21.15.129:0/1778736703 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f84030770 con 0x7f3f9415a2e0 2024-03-23T15:50:06.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.774+0000 7f3f8bfff640 1 -- 172.21.15.129:0/1778736703 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84030b00 con 0x7f3f9415a2e0 2024-03-23T15:50:06.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 -- 172.21.15.129:0/1778736703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 msgr2=0x7f3f9406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:06.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/1778736703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f9406bb70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3f840097a0 tx=0x7f3f8402b960 comp rx=0 tx=0).stop 2024-03-23T15:50:06.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 -- 172.21.15.129:0/1778736703 shutdown_connections 2024-03-23T15:50:06.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/1778736703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:06.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/1778736703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f9406bb70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:06.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 -- 172.21.15.129:0/1778736703 >> 172.21.15.129:0/1778736703 conn(0x7f3f94061b50 msgr2=0x7f3f94061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:06.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 -- 172.21.15.129:0/1778736703 shutdown_connections 2024-03-23T15:50:06.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.775+0000 7f3f9b42e640 1 -- 172.21.15.129:0/1778736703 wait complete. 2024-03-23T15:50:06.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.777+0000 7f3f9b42e640 1 Processor -- start 2024-03-23T15:50:06.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.777+0000 7f3f9b42e640 1 -- start start 2024-03-23T15:50:06.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.778+0000 7f3f9b42e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.778+0000 7f3f9b42e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f941544b0 0x7f3f94152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.778+0000 7f3f9b42e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f941f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.778+0000 7f3f9b42e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f940d0620 con 0x7f3f9415a2e0 2024-03-23T15:50:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.778+0000 7f3f9b42e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f940c9490 con 0x7f3f941544b0 2024-03-23T15:50:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.778+0000 7f3f9b42e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f940d0820 con 0x7f3f9406c0b0 2024-03-23T15:50:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.779+0000 7f3f999cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f941f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:06.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.779+0000 7f3f989c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f941544b0 0x7f3f94152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:06.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.779+0000 7f3f991ca640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:06.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.780+0000 7f3f999cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f941f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54120/0 (socket says 172.21.15.129:54120) 2024-03-23T15:50:06.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.780+0000 7f3f999cb640 1 -- 172.21.15.129:0/2226541529 learned_addr learned my addr 172.21.15.129:0/2226541529 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:06.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.783+0000 7f3f999cb640 1 -- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 msgr2=0x7f3f940d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:06.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.783+0000 7f3f999cb640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:06.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.783+0000 7f3f999cb640 1 -- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f941544b0 msgr2=0x7f3f94152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:06.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.783+0000 7f3f999cb640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f941544b0 0x7f3f94152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:06.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.783+0000 7f3f999cb640 1 -- 172.21.15.129:0/2226541529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f84009400 con 0x7f3f9415a2e0 2024-03-23T15:50:06.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.786+0000 7f3f999cb640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f941f35c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00efc0 tx=0x7f3f8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:06.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.791+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c014480 con 0x7f3f9415a2e0 2024-03-23T15:50:06.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.791+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f8c00f040 con 0x7f3f9415a2e0 2024-03-23T15:50:06.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.791+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f94156b00 con 0x7f3f9415a2e0 2024-03-23T15:50:06.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.791+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c014820 con 0x7f3f9415a2e0 2024-03-23T15:50:06.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.792+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f941570c0 con 0x7f3f9415a2e0 2024-03-23T15:50:06.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.793+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f941f32e0 con 0x7f3f9415a2e0 2024-03-23T15:50:06.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.817+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3f8c014980 con 0x7f3f9415a2e0 2024-03-23T15:50:06.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.817+0000 7f3f89ffb640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3f7806b810 0x7f3f7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:06.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.818+0000 7f3f991ca640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3f7806b810 0x7f3f7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.818+0000 7f3f991ca640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3f7806b810 0x7f3f7806dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f90005880 tx=0x7f3f90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:06.820+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3f8c059d90 con 0x7f3f9415a2e0 2024-03-23T15:50:07.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.102+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3f8c056030 con 0x7f3f9415a2e0 2024-03-23T15:50:07.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.331+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f94075740 con 0x7f3f9415a2e0 2024-03-23T15:50:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.420+0000 7f3f89ffb640 1 -- 172.21.15.129:0/2226541529 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3f8c018070 con 0x7f3f9415a2e0 2024-03-23T15:50:07.424 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:07.424 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:07.424 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3f7806b810 msgr2=0x7f3f7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3f7806b810 0x7f3f7806dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f90005880 tx=0x7f3f90009040 comp rx=0 tx=0).stop 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 msgr2=0x7f3f941f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f941f35c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00efc0 tx=0x7f3f8c00c490 comp rx=0 tx=0).stop 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 shutdown_connections 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f9415a2e0 0x7f3f941f35c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3f7806b810 0x7f3f7806dcd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f941544b0 0x7f3f94152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 --2- 172.21.15.129:0/2226541529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f9406c0b0 0x7f3f940d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 >> 172.21.15.129:0/2226541529 conn(0x7f3f94061b50 msgr2=0x7f3f940c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:07.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 shutdown_connections 2024-03-23T15:50:07.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:07.424+0000 7f3f9b42e640 1 -- 172.21.15.129:0/2226541529 wait complete. 2024-03-23T15:50:08.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:08.048 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:09.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.150+0000 7f2f359a1640 1 Processor -- start 2024-03-23T15:50:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.151+0000 7f2f359a1640 1 -- start start 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.151+0000 7f2f359a1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2f30072ad0 0x7f2f3006d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.151+0000 7f2f359a1640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f30073560 con 0x7f2f30150d00 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.151+0000 7f2f359a1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f30072fc0 con 0x7f2f300c7e70 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.151+0000 7f2f359a1640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f2f301510e0 con 0x7f2f30072ad0 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.152+0000 7f2f2e7fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2f30072ad0 0x7f2f3006d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.152+0000 7f2f2f7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2f300c7e70 0x7f2f30161190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:34632/0 (socket says 172.21.15.129:34632) 2024-03-23T15:50:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.152+0000 7f2f2f7fe640 1 -- 172.21.15.129:0/102070532 learned_addr learned my addr 172.21.15.129:0/102070532 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.156+0000 7f2f2dffb640 1 -- 172.21.15.129:0/102070532 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31593003 0 0) 0x7f2f30072fc0 con 0x7f2f300c7e70 2024-03-23T15:50:09.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.156+0000 7f2f2dffb640 1 -- 172.21.15.129:0/102070532 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f18003520 con 0x7f2f300c7e70 2024-03-23T15:50:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.156+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/102070532 >> v1:172.21.15.157:6789/0 conn(0x7f2f300c7e70 legacy=0x7f2f30161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.156+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/102070532 >> v1:172.21.15.129:6789/0 conn(0x7f2f30150d00 legacy=0x7f2f301568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.156+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/102070532 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f30073280 con 0x7f2f30072ad0 2024-03-23T15:50:09.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.160+0000 7f2f2e7fc640 1 --2- 172.21.15.129:0/102070532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f30072ad0 0x7f2f3006d9b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f240097a0 tx=0x7f2f2402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3afce40875ce594 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.164+0000 7f2f2dffb640 1 -- 172.21.15.129:0/102070532 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24030610 con 0x7f2f30072ad0 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.164+0000 7f2f2dffb640 1 -- 172.21.15.129:0/102070532 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f24030770 con 0x7f2f30072ad0 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.164+0000 7f2f2dffb640 1 -- 172.21.15.129:0/102070532 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24030b00 con 0x7f2f30072ad0 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 -- 172.21.15.129:0/102070532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f30072ad0 msgr2=0x7f2f3006d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 --2- 172.21.15.129:0/102070532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f30072ad0 0x7f2f3006d9b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f240097a0 tx=0x7f2f2402b960 comp rx=0 tx=0).stop 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 -- 172.21.15.129:0/102070532 shutdown_connections 2024-03-23T15:50:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 --2- 172.21.15.129:0/102070532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f30072ad0 0x7f2f3006d9b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 -- 172.21.15.129:0/102070532 >> 172.21.15.129:0/102070532 conn(0x7f2f300c8370 msgr2=0x7f2f300c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:09.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 -- 172.21.15.129:0/102070532 shutdown_connections 2024-03-23T15:50:09.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.165+0000 7f2f359a1640 1 -- 172.21.15.129:0/102070532 wait complete. 2024-03-23T15:50:09.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.166+0000 7f2f359a1640 1 Processor -- start 2024-03-23T15:50:09.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.166+0000 7f2f359a1640 1 -- start start 2024-03-23T15:50:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.167+0000 7f2f359a1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f30072ad0 0x7f2f301f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.167+0000 7f2f359a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 0x7f2f30158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.167+0000 7f2f359a1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f30150d00 0x7f2f30159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.167+0000 7f2f359a1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2f300c6790 con 0x7f2f300c7e70 2024-03-23T15:50:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.167+0000 7f2f359a1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2f301efc90 con 0x7f2f30072ad0 2024-03-23T15:50:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.167+0000 7f2f359a1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2f301eff50 con 0x7f2f30150d00 2024-03-23T15:50:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.168+0000 7f2f2f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f30150d00 0x7f2f30159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.168+0000 7f2f2e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 0x7f2f30158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.168+0000 7f2f2effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f30072ad0 0x7f2f301f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.168+0000 7f2f2e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 0x7f2f30158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54140/0 (socket says 172.21.15.129:54140) 2024-03-23T15:50:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.168+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/1765367247 learned_addr learned my addr 172.21.15.129:0/1765367247 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.173+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f30150d00 msgr2=0x7f2f30159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.173+0000 7f2f2e7fc640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f30150d00 0x7f2f30159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.173+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f30072ad0 msgr2=0x7f2f301f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.173+0000 7f2f2e7fc640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f30072ad0 0x7f2f301f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.173+0000 7f2f2e7fc640 1 -- 172.21.15.129:0/1765367247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f24009400 con 0x7f2f300c7e70 2024-03-23T15:50:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.177+0000 7f2f2e7fc640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 0x7f2f30158df0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098c0 tx=0x7f2f24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:09.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.182+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2403e040 con 0x7f2f300c7e70 2024-03-23T15:50:09.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.182+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f24039cb0 con 0x7f2f300c7e70 2024-03-23T15:50:09.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.182+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f301f0230 con 0x7f2f300c7e70 2024-03-23T15:50:09.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.182+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24044630 con 0x7f2f300c7e70 2024-03-23T15:50:09.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.182+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f301f07f0 con 0x7f2f300c7e70 2024-03-23T15:50:09.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.183+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f30075740 con 0x7f2f300c7e70 2024-03-23T15:50:09.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.223+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f2404d3b0 con 0x7f2f300c7e70 2024-03-23T15:50:09.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.223+0000 7f2f17fff640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2f1c06b890 0x7f2f1c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:09.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.224+0000 7f2f2effd640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2f1c06b890 0x7f2f1c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:09.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.224+0000 7f2f2effd640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2f1c06b890 0x7f2f1c06dd50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2f200016e0 tx=0x7f2f20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:09.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.484+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2f24039e20 con 0x7f2f300c7e70 2024-03-23T15:50:09.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.485+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f24077590 con 0x7f2f300c7e70 2024-03-23T15:50:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.730+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f300c43b0 con 0x7f2f300c7e70 2024-03-23T15:50:09.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.753+0000 7f2f17fff640 1 -- 172.21.15.129:0/1765367247 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2f24034090 con 0x7f2f300c7e70 2024-03-23T15:50:09.756 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:09.757 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2f1c06b890 msgr2=0x7f2f1c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2f1c06b890 0x7f2f1c06dd50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2f200016e0 tx=0x7f2f20009040 comp rx=0 tx=0).stop 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 msgr2=0x7f2f30158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 0x7f2f30158df0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098c0 tx=0x7f2f24004730 comp rx=0 tx=0).stop 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 shutdown_connections 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f30150d00 0x7f2f30159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f300c7e70 0x7f2f30158df0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2f1c06b890 0x7f2f1c06dd50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 --2- 172.21.15.129:0/1765367247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f30072ad0 0x7f2f301f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 >> 172.21.15.129:0/1765367247 conn(0x7f2f300c8370 msgr2=0x7f2f300c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:09.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.755+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 shutdown_connections 2024-03-23T15:50:09.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:09.756+0000 7f2f359a1640 1 -- 172.21.15.129:0/1765367247 wait complete. 2024-03-23T15:50:10.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:10.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:11.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:11.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.477+0000 7fbeb373a640 1 Processor -- start 2024-03-23T15:50:11.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.478+0000 7fbeb373a640 1 -- start start 2024-03-23T15:50:11.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.478+0000 7fbeb373a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeac1544b0 0x7fbeac15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:11.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.478+0000 7fbeb373a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbeac0caa00 0x7fbeac06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:11.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.478+0000 7fbeb373a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeac0cbcb0 con 0x7fbeac06c0b0 2024-03-23T15:50:11.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.478+0000 7fbeb373a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fbeac0c5be0 con 0x7fbeac1544b0 2024-03-23T15:50:11.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.478+0000 7fbeb373a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbeac15b280 con 0x7fbeac0caa00 2024-03-23T15:50:11.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.479+0000 7fbeb14d6640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeac1544b0 0x7fbeac15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:11.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.480+0000 7fbeb14d6640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbeac1544b0 0x7fbeac15a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37648/0 (socket says 172.21.15.129:37648) 2024-03-23T15:50:11.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.480+0000 7fbeb14d6640 1 -- 172.21.15.129:0/398294078 learned_addr learned my addr 172.21.15.129:0/398294078 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:11.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.480+0000 7fbeb0cd5640 1 --2- 172.21.15.129:0/398294078 >> v2:172.21.15.157:3300/0 conn(0x7fbeac0caa00 0x7fbeac06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.482+0000 7fbeb14d6640 1 -- 172.21.15.129:0/398294078 >> v2:172.21.15.157:3300/0 conn(0x7fbeac0caa00 msgr2=0x7fbeac06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.482+0000 7fbeb14d6640 1 --2- 172.21.15.129:0/398294078 >> v2:172.21.15.157:3300/0 conn(0x7fbeac0caa00 0x7fbeac06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.482+0000 7fbeb14d6640 1 -- 172.21.15.129:0/398294078 >> v1:172.21.15.157:6789/0 conn(0x7fbeac06c0b0 legacy=0x7fbeac0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:11.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.482+0000 7fbeb14d6640 1 -- 172.21.15.129:0/398294078 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeac15a9e0 con 0x7fbeac1544b0 2024-03-23T15:50:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.484+0000 7fbeb14d6640 1 --2- 172.21.15.129:0/398294078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac1544b0 0x7fbeac15a2e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbeac06ce60 tx=0x7fbea802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db7cee9060460ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:11.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.486+0000 7fbea3fff640 1 -- 172.21.15.129:0/398294078 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea802f850 con 0x7fbeac1544b0 2024-03-23T15:50:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.488+0000 7fbea3fff640 1 -- 172.21.15.129:0/398294078 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea802f9b0 con 0x7fbeac1544b0 2024-03-23T15:50:11.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.488+0000 7fbea3fff640 1 -- 172.21.15.129:0/398294078 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea802fd40 con 0x7fbeac1544b0 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.489+0000 7fbeb373a640 1 -- 172.21.15.129:0/398294078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac1544b0 msgr2=0x7fbeac15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.489+0000 7fbeb373a640 1 --2- 172.21.15.129:0/398294078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac1544b0 0x7fbeac15a2e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbeac06ce60 tx=0x7fbea802ba30 comp rx=0 tx=0).stop 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.490+0000 7fbeb373a640 1 -- 172.21.15.129:0/398294078 shutdown_connections 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.490+0000 7fbeb373a640 1 --2- 172.21.15.129:0/398294078 >> v2:172.21.15.157:3300/0 conn(0x7fbeac0caa00 0x7fbeac06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.490+0000 7fbeb373a640 1 --2- 172.21.15.129:0/398294078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac1544b0 0x7fbeac15a2e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.490+0000 7fbeb373a640 1 -- 172.21.15.129:0/398294078 >> 172.21.15.129:0/398294078 conn(0x7fbeac061b50 msgr2=0x7fbeac061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.490+0000 7fbeb373a640 1 -- 172.21.15.129:0/398294078 shutdown_connections 2024-03-23T15:50:11.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.490+0000 7fbeb373a640 1 -- 172.21.15.129:0/398294078 wait complete. 2024-03-23T15:50:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.491+0000 7fbeb373a640 1 Processor -- start 2024-03-23T15:50:11.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.491+0000 7fbeb373a640 1 -- start start 2024-03-23T15:50:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb373a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 0x7fbeac0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb373a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbeac0caa00 0x7fbeac0c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb373a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 0x7fbeac1f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb373a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbeac156540 con 0x7fbeac06c0b0 2024-03-23T15:50:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb373a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbeac0ceed0 con 0x7fbeac1544b0 2024-03-23T15:50:11.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb373a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbeac1f4110 con 0x7fbeac0caa00 2024-03-23T15:50:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb14d6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 0x7fbeac0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.492+0000 7fbeb1cd7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 0x7fbeac1f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.493+0000 7fbeb1cd7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 0x7fbeac1f0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45150/0 (socket says 172.21.15.129:45150) 2024-03-23T15:50:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.493+0000 7fbeb14d6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 0x7fbeac0ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37656/0 (socket says 172.21.15.129:37656) 2024-03-23T15:50:11.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.493+0000 7fbeb1cd7640 1 -- 172.21.15.129:0/2818989990 learned_addr learned my addr 172.21.15.129:0/2818989990 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:11.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.493+0000 7fbeb0cd5640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbeac0caa00 0x7fbeac0c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.496+0000 7fbeb1cd7640 1 -- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbeac0caa00 msgr2=0x7fbeac0c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.496+0000 7fbeb1cd7640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbeac0caa00 0x7fbeac0c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.496+0000 7fbeb1cd7640 1 -- 172.21.15.129:0/2818989990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 msgr2=0x7fbeac0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.496+0000 7fbeb1cd7640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 0x7fbeac0ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.496+0000 7fbeb1cd7640 1 -- 172.21.15.129:0/2818989990 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea80094d0 con 0x7fbeac1544b0 2024-03-23T15:50:11.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.496+0000 7fbeb14d6640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 0x7fbeac0ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:50:11.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.498+0000 7fbeb1cd7640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 0x7fbeac1f0010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbea400c7d0 tx=0x7fbea400cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:11.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.503+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4003b70 con 0x7fbeac1544b0 2024-03-23T15:50:11.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.503+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea4003cf0 con 0x7fbeac1544b0 2024-03-23T15:50:11.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.503+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea400aa90 con 0x7fbeac1544b0 2024-03-23T15:50:11.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.503+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeac156a10 con 0x7fbeac1544b0 2024-03-23T15:50:11.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.503+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeac0cec60 con 0x7fbeac1544b0 2024-03-23T15:50:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.504+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeac075740 con 0x7fbeac1544b0 2024-03-23T15:50:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.535+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbea400abf0 con 0x7fbeac1544b0 2024-03-23T15:50:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.535+0000 7fbea1ffb640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe9006b810 0x7fbe9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.535+0000 7fbeb14d6640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe9006b810 0x7fbe9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.536+0000 7fbeb14d6640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe9006b810 0x7fbe9006dcd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbea8036000 tx=0x7fbea8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.540+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fbea4059210 con 0x7fbeac1544b0 2024-03-23T15:50:11.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:11.789+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbea40554b0 con 0x7fbeac1544b0 2024-03-23T15:50:12.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.008+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbeac1f4250 con 0x7fbeac1544b0 2024-03-23T15:50:12.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.020+0000 7fbea1ffb640 1 -- 172.21.15.129:0/2818989990 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fbea4003e60 con 0x7fbeac1544b0 2024-03-23T15:50:12.023 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:12.024 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.022+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe9006b810 msgr2=0x7fbe9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.022+0000 7fbeb373a640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe9006b810 0x7fbe9006dcd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbea8036000 tx=0x7fbea8008030 comp rx=0 tx=0).stop 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.022+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 msgr2=0x7fbeac1f0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.022+0000 7fbeb373a640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 0x7fbeac1f0010 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbea400c7d0 tx=0x7fbea400cc90 comp rx=0 tx=0).stop 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 shutdown_connections 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe9006b810 0x7fbe9006dcd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbeac1544b0 0x7fbeac1f0010 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbeac0caa00 0x7fbeac0c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 --2- 172.21.15.129:0/2818989990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbeac06c0b0 0x7fbeac0ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 >> 172.21.15.129:0/2818989990 conn(0x7fbeac061b50 msgr2=0x7fbeac0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 shutdown_connections 2024-03-23T15:50:12.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:12.023+0000 7fbeb373a640 1 -- 172.21.15.129:0/2818989990 wait complete. 2024-03-23T15:50:12.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:12.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:13.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:13.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.748+0000 7f5c10aed640 1 Processor -- start 2024-03-23T15:50:13.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.748+0000 7f5c10aed640 1 -- start start 2024-03-23T15:50:13.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.749+0000 7f5c10aed640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c0c15a2e0 0x7f5c0c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:13.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.749+0000 7f5c10aed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0cbcb0 con 0x7f5c0c1544b0 2024-03-23T15:50:13.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.749+0000 7f5c10aed640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c15b280 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.749+0000 7f5c10aed640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5c0c0c5be0 con 0x7f5c0c15a2e0 2024-03-23T15:50:13.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.750+0000 7f5c0a59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f5c0c1544b0 0x7f5c0c0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:40222/0 (socket says 172.21.15.129:40222) 2024-03-23T15:50:13.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.750+0000 7f5c0a59c640 1 -- 172.21.15.129:0/3177732780 learned_addr learned my addr 172.21.15.129:0/3177732780 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:13.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.750+0000 7f5c09d9b640 1 --2- 172.21.15.129:0/3177732780 >> v2:172.21.15.157:3300/0 conn(0x7f5c0c15a2e0 0x7f5c0c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:13.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.753+0000 7f5c0959a640 1 -- 172.21.15.129:0/3177732780 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718923588 0 0) 0x7f5c0c0cbcb0 con 0x7f5c0c1544b0 2024-03-23T15:50:13.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.753+0000 7f5c0959a640 1 -- 172.21.15.129:0/3177732780 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf4003520 con 0x7f5c0c1544b0 2024-03-23T15:50:13.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.756+0000 7f5c09d9b640 1 -- 172.21.15.129:0/3177732780 >> v1:172.21.15.157:6789/0 conn(0x7f5c0c06c0b0 legacy=0x7f5c0c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:13.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.756+0000 7f5c09d9b640 1 -- 172.21.15.129:0/3177732780 >> v1:172.21.15.129:6789/0 conn(0x7f5c0c1544b0 legacy=0x7f5c0c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:13.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.756+0000 7f5c09d9b640 1 -- 172.21.15.129:0/3177732780 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c15a9e0 con 0x7f5c0c15a2e0 2024-03-23T15:50:13.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.760+0000 7f5c09d9b640 1 --2- 172.21.15.129:0/3177732780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c15a2e0 0x7f5c0c06bb70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5c000097a0 tx=0x7f5c0002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bed4b051ffb7bea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:13.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c0959a640 1 -- 172.21.15.129:0/3177732780 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0002f810 con 0x7f5c0c15a2e0 2024-03-23T15:50:13.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c0959a640 1 -- 172.21.15.129:0/3177732780 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c0002f970 con 0x7f5c0c15a2e0 2024-03-23T15:50:13.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c0959a640 1 -- 172.21.15.129:0/3177732780 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0002fd00 con 0x7f5c0c15a2e0 2024-03-23T15:50:13.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 -- 172.21.15.129:0/3177732780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c15a2e0 msgr2=0x7f5c0c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:13.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 --2- 172.21.15.129:0/3177732780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c15a2e0 0x7f5c0c06bb70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5c000097a0 tx=0x7f5c0002b9f0 comp rx=0 tx=0).stop 2024-03-23T15:50:13.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 -- 172.21.15.129:0/3177732780 shutdown_connections 2024-03-23T15:50:13.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 --2- 172.21.15.129:0/3177732780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c15a2e0 0x7f5c0c06bb70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:13.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 -- 172.21.15.129:0/3177732780 >> 172.21.15.129:0/3177732780 conn(0x7f5c0c061b50 msgr2=0x7f5c0c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:13.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 -- 172.21.15.129:0/3177732780 shutdown_connections 2024-03-23T15:50:13.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.765+0000 7f5c10aed640 1 -- 172.21.15.129:0/3177732780 wait complete. 2024-03-23T15:50:13.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.767+0000 7f5c10aed640 1 Processor -- start 2024-03-23T15:50:13.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.767+0000 7f5c10aed640 1 -- start start 2024-03-23T15:50:13.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.767+0000 7f5c10aed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 0x7f5c0c141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:13.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c10aed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c0c1544b0 0x7f5c0c1f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:13.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c10aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c0c15a2e0 0x7f5c0c159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:13.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c10aed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c0cf820 con 0x7f5c0c1544b0 2024-03-23T15:50:13.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c10aed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c154f60 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c10aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c0cf390 con 0x7f5c0c15a2e0 2024-03-23T15:50:13.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c09d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c0c1544b0 0x7f5c0c1f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:13.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c0a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 0x7f5c0c141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:13.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c0ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c0c15a2e0 0x7f5c0c159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:13.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c0a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 0x7f5c0c141980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45176/0 (socket says 172.21.15.129:45176) 2024-03-23T15:50:13.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c09d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c0c1544b0 0x7f5c0c1f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37662/0 (socket says 172.21.15.129:37662) 2024-03-23T15:50:13.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.768+0000 7f5c0a59c640 1 -- 172.21.15.129:0/1749775418 learned_addr learned my addr 172.21.15.129:0/1749775418 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.772+0000 7f5c0a59c640 1 -- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c0c15a2e0 msgr2=0x7f5c0c159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.772+0000 7f5c0a59c640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c0c15a2e0 0x7f5c0c159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.772+0000 7f5c0a59c640 1 -- 172.21.15.129:0/1749775418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c0c1544b0 msgr2=0x7f5c0c1f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.772+0000 7f5c0a59c640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c0c1544b0 0x7f5c0c1f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.772+0000 7f5c0a59c640 1 -- 172.21.15.129:0/1749775418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c00009400 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.774+0000 7f5c0a59c640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 0x7f5c0c141980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00c530 tx=0x7f5bfc00c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:13.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.778+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc005ce0 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.778+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bfc005e40 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.778+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c14adc0 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.778+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc0072b0 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.779+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c0ced40 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.779+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c0c075740 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.800+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5bfc007410 con 0x7f5c0c06c0b0 2024-03-23T15:50:13.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.800+0000 7f5bfaffd640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bec06b890 0x7f5bec06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:13.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.800+0000 7f5c09d9b640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bec06b890 0x7f5bec06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:13.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:13.801+0000 7f5c09d9b640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bec06b890 0x7f5bec06dd50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c00036000 tx=0x7f5c00008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:14.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.044+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5bfc058470 con 0x7f5c0c06c0b0 2024-03-23T15:50:14.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.054+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5bfc054710 con 0x7f5c0c06c0b0 2024-03-23T15:50:14.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.275+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c0c157d00 con 0x7f5c0c06c0b0 2024-03-23T15:50:14.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.290+0000 7f5bfaffd640 1 -- 172.21.15.129:0/1749775418 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5bfc017020 con 0x7f5c0c06c0b0 2024-03-23T15:50:14.293 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:14.293 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:14.294 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:14.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bec06b890 msgr2=0x7f5bec06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:14.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bec06b890 0x7f5bec06dd50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c00036000 tx=0x7f5c00008030 comp rx=0 tx=0).stop 2024-03-23T15:50:14.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 msgr2=0x7f5c0c141980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:14.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 0x7f5c0c141980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00c530 tx=0x7f5bfc00c9f0 comp rx=0 tx=0).stop 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 shutdown_connections 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c0c15a2e0 0x7f5c0c159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c0c1544b0 0x7f5c0c1f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bec06b890 0x7f5bec06dd50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 --2- 172.21.15.129:0/1749775418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c0c06c0b0 0x7f5c0c141980 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 >> 172.21.15.129:0/1749775418 conn(0x7f5c0c061b50 msgr2=0x7f5c0c0b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 shutdown_connections 2024-03-23T15:50:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:14.292+0000 7f5c10aed640 1 -- 172.21.15.129:0/1749775418 wait complete. 2024-03-23T15:50:14.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:14.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:15.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:16.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.011+0000 7f62f3d93640 1 Processor -- start 2024-03-23T15:50:16.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.011+0000 7f62f3d93640 1 -- start start 2024-03-23T15:50:16.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f3d93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec1544b0 0x7f62ec15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:16.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f3d93640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f62ec0caa00 0x7f62ec06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:16.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f3d93640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec0cbcb0 con 0x7f62ec06c0b0 2024-03-23T15:50:16.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f3d93640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec0c5be0 con 0x7f62ec0caa00 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f3d93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec15b280 con 0x7f62ec1544b0 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f132e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f62ec0caa00 0x7f62ec06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.012+0000 7f62f1b2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec1544b0 0x7f62ec15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.013+0000 7f62f132e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f62ec0caa00 0x7f62ec06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37674/0 (socket says 172.21.15.129:37674) 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.013+0000 7f62f132e640 1 -- 172.21.15.129:0/1897755132 learned_addr learned my addr 172.21.15.129:0/1897755132 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.016+0000 7f62f1b2f640 1 -- 172.21.15.129:0/1897755132 >> v2:172.21.15.129:3300/0 conn(0x7f62ec0caa00 msgr2=0x7f62ec06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:16.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.016+0000 7f62f1b2f640 1 --2- 172.21.15.129:0/1897755132 >> v2:172.21.15.129:3300/0 conn(0x7f62ec0caa00 0x7f62ec06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.016+0000 7f62f1b2f640 1 -- 172.21.15.129:0/1897755132 >> v1:172.21.15.129:6789/0 conn(0x7f62ec06c0b0 legacy=0x7f62ec0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:16.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.016+0000 7f62f1b2f640 1 -- 172.21.15.129:0/1897755132 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec15a9e0 con 0x7f62ec1544b0 2024-03-23T15:50:16.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.019+0000 7f62f1b2f640 1 --2- 172.21.15.129:0/1897755132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec1544b0 0x7f62ec15a2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62e80097a0 tx=0x7f62e802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7b24b9b363deb9a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:16.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.024+0000 7f62f0b2d640 1 -- 172.21.15.129:0/1897755132 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8030610 con 0x7f62ec1544b0 2024-03-23T15:50:16.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.024+0000 7f62f0b2d640 1 -- 172.21.15.129:0/1897755132 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e8030770 con 0x7f62ec1544b0 2024-03-23T15:50:16.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.024+0000 7f62f0b2d640 1 -- 172.21.15.129:0/1897755132 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8030b00 con 0x7f62ec1544b0 2024-03-23T15:50:16.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 -- 172.21.15.129:0/1897755132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec1544b0 msgr2=0x7f62ec15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:16.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 --2- 172.21.15.129:0/1897755132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec1544b0 0x7f62ec15a2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62e80097a0 tx=0x7f62e802b960 comp rx=0 tx=0).stop 2024-03-23T15:50:16.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 -- 172.21.15.129:0/1897755132 shutdown_connections 2024-03-23T15:50:16.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 --2- 172.21.15.129:0/1897755132 >> v2:172.21.15.129:3300/0 conn(0x7f62ec0caa00 0x7f62ec06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 --2- 172.21.15.129:0/1897755132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec1544b0 0x7f62ec15a2e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 -- 172.21.15.129:0/1897755132 >> 172.21.15.129:0/1897755132 conn(0x7f62ec061b50 msgr2=0x7f62ec061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:16.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 -- 172.21.15.129:0/1897755132 shutdown_connections 2024-03-23T15:50:16.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.025+0000 7f62f3d93640 1 -- 172.21.15.129:0/1897755132 wait complete. 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.026+0000 7f62f3d93640 1 Processor -- start 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.026+0000 7f62f3d93640 1 -- start start 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.027+0000 7f62f3d93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 0x7f62ec0ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.027+0000 7f62f3d93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec0caa00 0x7f62ec159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.027+0000 7f62f3d93640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ec1544b0 0x7f62ec1f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.027+0000 7f62f3d93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0d03f0 con 0x7f62ec06c0b0 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.027+0000 7f62f3d93640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec1f3560 con 0x7f62ec1544b0 2024-03-23T15:50:16.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.027+0000 7f62f3d93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec156e60 con 0x7f62ec0caa00 2024-03-23T15:50:16.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.028+0000 7f62f132e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec0caa00 0x7f62ec159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:16.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.028+0000 7f62f1b2f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 0x7f62ec0ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:16.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.028+0000 7f62f2330640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ec1544b0 0x7f62ec1f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:16.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.028+0000 7f62f1b2f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 0x7f62ec0ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37684/0 (socket says 172.21.15.129:37684) 2024-03-23T15:50:16.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.028+0000 7f62f1b2f640 1 -- 172.21.15.129:0/2591008801 learned_addr learned my addr 172.21.15.129:0/2591008801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:16.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.031+0000 7f62f1b2f640 1 -- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec0caa00 msgr2=0x7f62ec159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:16.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.031+0000 7f62f1b2f640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec0caa00 0x7f62ec159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.031+0000 7f62f1b2f640 1 -- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ec1544b0 msgr2=0x7f62ec1f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:16.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.031+0000 7f62f1b2f640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ec1544b0 0x7f62ec1f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.031+0000 7f62f1b2f640 1 -- 172.21.15.129:0/2591008801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e8009400 con 0x7f62ec06c0b0 2024-03-23T15:50:16.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.034+0000 7f62f1b2f640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 0x7f62ec0ce1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f62e80098c0 tx=0x7f62e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:16.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.036+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e803e070 con 0x7f62ec06c0b0 2024-03-23T15:50:16.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.036+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec1efff0 con 0x7f62ec06c0b0 2024-03-23T15:50:16.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.036+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1f0550 con 0x7f62ec06c0b0 2024-03-23T15:50:16.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.037+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec075740 con 0x7f62ec06c0b0 2024-03-23T15:50:16.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.040+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e8030e10 con 0x7f62ec06c0b0 2024-03-23T15:50:16.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.040+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8039c10 con 0x7f62ec06c0b0 2024-03-23T15:50:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.329+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f62e80423f0 con 0x7f62ec06c0b0 2024-03-23T15:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.330+0000 7f62e27fc640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f62d006b810 0x7f62d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.330+0000 7f62f132e640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f62d006b810 0x7f62d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.331+0000 7f62f132e640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f62d006b810 0x7f62d006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f62ec153850 tx=0x7f62dc009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.354+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f62e807ab90 con 0x7f62ec06c0b0 2024-03-23T15:50:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.354+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f62e8076e70 con 0x7f62ec06c0b0 2024-03-23T15:50:16.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.613+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62ec0c9f30 con 0x7f62ec06c0b0 2024-03-23T15:50:16.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.635+0000 7f62e27fc640 1 -- 172.21.15.129:0/2591008801 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f62e8048090 con 0x7f62ec06c0b0 2024-03-23T15:50:16.638 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:16.638 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:16.639 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f62d006b810 msgr2=0x7f62d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f62d006b810 0x7f62d006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f62ec153850 tx=0x7f62dc009240 comp rx=0 tx=0).stop 2024-03-23T15:50:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 msgr2=0x7f62ec0ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 0x7f62ec0ce1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f62e80098c0 tx=0x7f62e80046c0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 shutdown_connections 2024-03-23T15:50:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f62d006b810 0x7f62d006dcd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62ec1544b0 0x7f62ec1f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62ec0caa00 0x7f62ec159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 --2- 172.21.15.129:0/2591008801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62ec06c0b0 0x7f62ec0ce1a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 >> 172.21.15.129:0/2591008801 conn(0x7f62ec061b50 msgr2=0x7f62ec14a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 shutdown_connections 2024-03-23T15:50:16.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:16.637+0000 7f62f3d93640 1 -- 172.21.15.129:0/2591008801 wait complete. 2024-03-23T15:50:17.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:17.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:18.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:18.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.360+0000 7fc5d82d4640 1 Processor -- start 2024-03-23T15:50:18.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.361+0000 7fc5d82d4640 1 -- start start 2024-03-23T15:50:18.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.361+0000 7fc5d82d4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5d01544b0 0x7fc5d015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:18.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.361+0000 7fc5d82d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d00c6000 0x7fc5d00c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.361+0000 7fc5d82d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d00cbcb0 con 0x7fc5d00caa00 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.361+0000 7fc5d82d4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d00c5be0 con 0x7fc5d01544b0 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.361+0000 7fc5d82d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d00c69b0 con 0x7fc5d00c6000 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.362+0000 7fc5d6070640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc5d01544b0 0x7fc5d015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.362+0000 7fc5d586f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc5d00caa00 0x7fc5d006bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:40852/0 (socket says 172.21.15.129:40852) 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.362+0000 7fc5d586f640 1 -- 172.21.15.129:0/345590410 learned_addr learned my addr 172.21.15.129:0/345590410 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:18.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.362+0000 7fc5d6871640 1 --2- 172.21.15.129:0/345590410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d00c6000 0x7fc5d00c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:18.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.365+0000 7fc5d6070640 1 -- 172.21.15.129:0/345590410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d00c6000 msgr2=0x7fc5d00c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.365+0000 7fc5d6070640 1 --2- 172.21.15.129:0/345590410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d00c6000 0x7fc5d00c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.365+0000 7fc5d6070640 1 -- 172.21.15.129:0/345590410 >> v1:172.21.15.157:6789/0 conn(0x7fc5d00caa00 legacy=0x7fc5d006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.365+0000 7fc5d6070640 1 -- 172.21.15.129:0/345590410 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d015b280 con 0x7fc5d01544b0 2024-03-23T15:50:18.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.367+0000 7fc5d6070640 1 --2- 172.21.15.129:0/345590410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d01544b0 0x7fc5d015a2e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0097a0 tx=0x7fc5cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531c34cd752dc7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.371+0000 7fc5d506e640 1 -- 172.21.15.129:0/345590410 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc030610 con 0x7fc5d01544b0 2024-03-23T15:50:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.371+0000 7fc5d506e640 1 -- 172.21.15.129:0/345590410 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5cc030770 con 0x7fc5d01544b0 2024-03-23T15:50:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.371+0000 7fc5d506e640 1 -- 172.21.15.129:0/345590410 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc030b00 con 0x7fc5d01544b0 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 -- 172.21.15.129:0/345590410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d01544b0 msgr2=0x7fc5d015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/345590410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d01544b0 0x7fc5d015a2e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0097a0 tx=0x7fc5cc02b960 comp rx=0 tx=0).stop 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 -- 172.21.15.129:0/345590410 shutdown_connections 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/345590410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d00c6000 0x7fc5d00c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/345590410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d01544b0 0x7fc5d015a2e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 -- 172.21.15.129:0/345590410 >> 172.21.15.129:0/345590410 conn(0x7fc5d0061b50 msgr2=0x7fc5d0061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 -- 172.21.15.129:0/345590410 shutdown_connections 2024-03-23T15:50:18.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.372+0000 7fc5d82d4640 1 -- 172.21.15.129:0/345590410 wait complete. 2024-03-23T15:50:18.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.373+0000 7fc5d82d4640 1 Processor -- start 2024-03-23T15:50:18.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.373+0000 7fc5d82d4640 1 -- start start 2024-03-23T15:50:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.374+0000 7fc5d82d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 0x7fc5d00ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.374+0000 7fc5d82d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5d00caa00 0x7fc5d00cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.374+0000 7fc5d82d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 0x7fc5d0158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.374+0000 7fc5d82d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d00cf490 con 0x7fc5d00c6000 2024-03-23T15:50:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.374+0000 7fc5d82d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d01573b0 con 0x7fc5d00caa00 2024-03-23T15:50:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.374+0000 7fc5d82d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d01584a0 con 0x7fc5d01544b0 2024-03-23T15:50:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d586f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5d00caa00 0x7fc5d00cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d6871640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 0x7fc5d0158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d6070640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 0x7fc5d00ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:18.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d6070640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 0x7fc5d00ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37706/0 (socket says 172.21.15.129:37706) 2024-03-23T15:50:18.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d6871640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 0x7fc5d0158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:41684/0 (socket says 172.21.15.129:41684) 2024-03-23T15:50:18.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d6070640 1 -- 172.21.15.129:0/1110440436 learned_addr learned my addr 172.21.15.129:0/1110440436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:18.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.375+0000 7fc5d586f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5d00caa00 0x7fc5d00cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45208/0 (socket says 172.21.15.129:45208) 2024-03-23T15:50:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.380+0000 7fc5d6871640 1 -- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5d00caa00 msgr2=0x7fc5d00cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.380+0000 7fc5d6871640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5d00caa00 0x7fc5d00cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.380+0000 7fc5d6871640 1 -- 172.21.15.129:0/1110440436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 msgr2=0x7fc5d00ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.380+0000 7fc5d6871640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 0x7fc5d00ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.380+0000 7fc5d6871640 1 -- 172.21.15.129:0/1110440436 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc009400 con 0x7fc5d01544b0 2024-03-23T15:50:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.380+0000 7fc5d6070640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 0x7fc5d00ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:50:18.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.384+0000 7fc5d6871640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 0x7fc5d0158960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800efc0 tx=0x7fc5c800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:18.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.388+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8009c00 con 0x7fc5d01544b0 2024-03-23T15:50:18.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.388+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c800f040 con 0x7fc5d01544b0 2024-03-23T15:50:18.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.388+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d015a220 con 0x7fc5d01544b0 2024-03-23T15:50:18.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.388+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80146c0 con 0x7fc5d01544b0 2024-03-23T15:50:18.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.388+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d0157620 con 0x7fc5d01544b0 2024-03-23T15:50:18.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.389+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d0075740 con 0x7fc5d01544b0 2024-03-23T15:50:18.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.419+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc5c8014820 con 0x7fc5d01544b0 2024-03-23T15:50:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.420+0000 7fc5c6ffd640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc5b406b810 0x7fc5b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.420+0000 7fc5d6070640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc5b406b810 0x7fc5b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:18.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.420+0000 7fc5d6070640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc5b406b810 0x7fc5b406dcd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc02b9a0 tx=0x7fc5cc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.663+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc5c8059d60 con 0x7fc5d01544b0 2024-03-23T15:50:18.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.684+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc5c8056000 con 0x7fc5d01544b0 2024-03-23T15:50:18.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.905+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5d0157d00 con 0x7fc5d01544b0 2024-03-23T15:50:18.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.924+0000 7fc5c6ffd640 1 -- 172.21.15.129:0/1110440436 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc5c8018090 con 0x7fc5d01544b0 2024-03-23T15:50:18.927 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:18.927 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:18.928 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc5b406b810 msgr2=0x7fc5b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc5b406b810 0x7fc5b406dcd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc02b9a0 tx=0x7fc5cc03b040 comp rx=0 tx=0).stop 2024-03-23T15:50:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 msgr2=0x7fc5d0158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 0x7fc5d0158960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800efc0 tx=0x7fc5c800c490 comp rx=0 tx=0).stop 2024-03-23T15:50:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 shutdown_connections 2024-03-23T15:50:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5d01544b0 0x7fc5d0158960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc5b406b810 0x7fc5b406dcd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5d00caa00 0x7fc5d00cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 --2- 172.21.15.129:0/1110440436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5d00c6000 0x7fc5d00ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:18.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 >> 172.21.15.129:0/1110440436 conn(0x7fc5d0061b50 msgr2=0x7fc5d014a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:18.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 shutdown_connections 2024-03-23T15:50:18.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:18.927+0000 7fc5d82d4640 1 -- 172.21.15.129:0/1110440436 wait complete. 2024-03-23T15:50:19.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:19.545 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:20.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:20.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.645+0000 7fe206815640 1 Processor -- start 2024-03-23T15:50:20.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.645+0000 7fe206815640 1 -- start start 2024-03-23T15:50:20.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe206815640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2000ca580 0x7fe20006c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:20.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe206815640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe20006c650 0x7fe2000c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:20.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe206815640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe20015d880 con 0x7fe200154270 2024-03-23T15:50:20.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe206815640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe20006ca60 con 0x7fe20006c650 2024-03-23T15:50:20.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe206815640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2000ca9f0 con 0x7fe2000ca580 2024-03-23T15:50:20.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe1ff7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2000ca580 0x7fe20006c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:20.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.646+0000 7fe204db2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe20006c650 0x7fe2000c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:20.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.647+0000 7fe1fffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe200154270 0x7fe20015ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33258/0 (socket says 172.21.15.129:33258) 2024-03-23T15:50:20.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.647+0000 7fe1fffff640 1 -- 172.21.15.129:0/3628347944 learned_addr learned my addr 172.21.15.129:0/3628347944 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:20.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.649+0000 7fe204db2640 1 -- 172.21.15.129:0/3628347944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2000ca580 msgr2=0x7fe20006c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:20.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.649+0000 7fe204db2640 1 --2- 172.21.15.129:0/3628347944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2000ca580 0x7fe20006c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:20.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.649+0000 7fe204db2640 1 -- 172.21.15.129:0/3628347944 >> v1:172.21.15.157:6789/0 conn(0x7fe200154270 legacy=0x7fe20015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:20.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.649+0000 7fe204db2640 1 -- 172.21.15.129:0/3628347944 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2000c4ae0 con 0x7fe20006c650 2024-03-23T15:50:20.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.651+0000 7fe204db2640 1 --2- 172.21.15.129:0/3628347944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000c4f70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009d90 tx=0x7fe1f802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13c8091714b05ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:20.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.655+0000 7fe1feffd640 1 -- 172.21.15.129:0/3628347944 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f802cb30 con 0x7fe20006c650 2024-03-23T15:50:20.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.655+0000 7fe1feffd640 1 -- 172.21.15.129:0/3628347944 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f802cc90 con 0x7fe20006c650 2024-03-23T15:50:20.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.655+0000 7fe1feffd640 1 -- 172.21.15.129:0/3628347944 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8034440 con 0x7fe20006c650 2024-03-23T15:50:20.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 -- 172.21.15.129:0/3628347944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 msgr2=0x7fe2000c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:20.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 --2- 172.21.15.129:0/3628347944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000c4f70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009d90 tx=0x7fe1f802bd80 comp rx=0 tx=0).stop 2024-03-23T15:50:20.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 -- 172.21.15.129:0/3628347944 shutdown_connections 2024-03-23T15:50:20.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 --2- 172.21.15.129:0/3628347944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000c4f70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:20.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 --2- 172.21.15.129:0/3628347944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2000ca580 0x7fe20006c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:20.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 -- 172.21.15.129:0/3628347944 >> 172.21.15.129:0/3628347944 conn(0x7fe20006d290 msgr2=0x7fe200072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:20.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 -- 172.21.15.129:0/3628347944 shutdown_connections 2024-03-23T15:50:20.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.656+0000 7fe206815640 1 -- 172.21.15.129:0/3628347944 wait complete. 2024-03-23T15:50:20.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.658+0000 7fe206815640 1 Processor -- start 2024-03-23T15:50:20.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.658+0000 7fe206815640 1 -- start start 2024-03-23T15:50:20.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.659+0000 7fe206815640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.659+0000 7fe206815640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2000ca580 0x7fe200152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.659+0000 7fe206815640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe200154270 0x7fe200153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.659+0000 7fe206815640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe200143430 con 0x7fe20006c650 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.659+0000 7fe206815640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001f4250 con 0x7fe2000ca580 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.659+0000 7fe206815640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2001f3640 con 0x7fe200154270 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.660+0000 7fe1ff7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2000ca580 0x7fe200152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.660+0000 7fe1fffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.660+0000 7fe204db2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe200154270 0x7fe200153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.660+0000 7fe1fffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000d0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46446/0 (socket says 172.21.15.129:46446) 2024-03-23T15:50:20.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.660+0000 7fe1fffff640 1 -- 172.21.15.129:0/2040426944 learned_addr learned my addr 172.21.15.129:0/2040426944 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:20.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.664+0000 7fe1fffff640 1 -- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe200154270 msgr2=0x7fe200153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:20.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.664+0000 7fe1fffff640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe200154270 0x7fe200153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:20.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.664+0000 7fe1fffff640 1 -- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2000ca580 msgr2=0x7fe200152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:20.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.664+0000 7fe1fffff640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2000ca580 0x7fe200152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:20.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.664+0000 7fe1fffff640 1 -- 172.21.15.129:0/2040426944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f8009a70 con 0x7fe20006c650 2024-03-23T15:50:20.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.666+0000 7fe1fffff640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000d0570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000c4a0 tx=0x7fe1f000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:20.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.669+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0005ce0 con 0x7fe20006c650 2024-03-23T15:50:20.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.669+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f0005e40 con 0x7fe20006c650 2024-03-23T15:50:20.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.669+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2001f3920 con 0x7fe20006c650 2024-03-23T15:50:20.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.669+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0007270 con 0x7fe20006c650 2024-03-23T15:50:20.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.669+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2001f3ee0 con 0x7fe20006c650 2024-03-23T15:50:20.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.671+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe200075740 con 0x7fe20006c650 2024-03-23T15:50:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.923+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe1f0007490 con 0x7fe20006c650 2024-03-23T15:50:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.924+0000 7fe1fcff9640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe1ec06b810 0x7fe1ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.924+0000 7fe1ff7fe640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe1ec06b810 0x7fe1ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.924+0000 7fe1ff7fe640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe1ec06b810 0x7fe1ec06dcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe2000cdb50 tx=0x7fe1f4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:20.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.938+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe1f0058350 con 0x7fe20006c650 2024-03-23T15:50:20.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:20.938+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe1f00545f0 con 0x7fe20006c650 2024-03-23T15:50:21.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.164+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe200153a90 con 0x7fe20006c650 2024-03-23T15:50:21.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.176+0000 7fe1fcff9640 1 -- 172.21.15.129:0/2040426944 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe1f0017090 con 0x7fe20006c650 2024-03-23T15:50:21.179 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:21.180 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:21.180 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:21.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe1ec06b810 msgr2=0x7fe1ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:21.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe1ec06b810 0x7fe1ec06dcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe2000cdb50 tx=0x7fe1f4009190 comp rx=0 tx=0).stop 2024-03-23T15:50:21.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 msgr2=0x7fe2000d0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000d0570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000c4a0 tx=0x7fe1f000c960 comp rx=0 tx=0).stop 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 shutdown_connections 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe200154270 0x7fe200153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe1ec06b810 0x7fe1ec06dcd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2000ca580 0x7fe200152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 --2- 172.21.15.129:0/2040426944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe20006c650 0x7fe2000d0570 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.179+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 >> 172.21.15.129:0/2040426944 conn(0x7fe20006d290 msgr2=0x7fe200072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.180+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 shutdown_connections 2024-03-23T15:50:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:21.180+0000 7fe206815640 1 -- 172.21.15.129:0/2040426944 wait complete. 2024-03-23T15:50:21.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:21.798 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:22.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:22.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.898+0000 7f23a3319640 1 Processor -- start 2024-03-23T15:50:22.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.899+0000 7f23a3319640 1 -- start start 2024-03-23T15:50:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.899+0000 7f23a3319640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f239c06c410 0x7f239c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.899+0000 7f23a3319640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c06c830 con 0x7f239c15de40 2024-03-23T15:50:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.899+0000 7f23a3319640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c0cc370 con 0x7f239c06c030 2024-03-23T15:50:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.899+0000 7f23a3319640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f239c06cae0 con 0x7f239c06c410 2024-03-23T15:50:22.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.900+0000 7f23a08b4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f239c06c410 0x7f239c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:22.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.900+0000 7f23a10b5640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f239c06c030 0x7f239c0cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33268/0 (socket says 172.21.15.129:33268) 2024-03-23T15:50:22.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.900+0000 7f23a10b5640 1 -- 172.21.15.129:0/1941020390 learned_addr learned my addr 172.21.15.129:0/1941020390 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:22.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.903+0000 7f2393fff640 1 -- 172.21.15.129:0/1941020390 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565282180 0 0) 0x7f239c0cc370 con 0x7f239c06c030 2024-03-23T15:50:22.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.903+0000 7f2393fff640 1 -- 172.21.15.129:0/1941020390 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2388003520 con 0x7f239c06c030 2024-03-23T15:50:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.904+0000 7f23a08b4640 1 -- 172.21.15.129:0/1941020390 >> v1:172.21.15.157:6789/0 conn(0x7f239c06c030 legacy=0x7f239c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.904+0000 7f23a08b4640 1 -- 172.21.15.129:0/1941020390 >> v1:172.21.15.129:6789/0 conn(0x7f239c15de40 legacy=0x7f239c164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.904+0000 7f23a08b4640 1 -- 172.21.15.129:0/1941020390 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c0cabc0 con 0x7f239c06c410 2024-03-23T15:50:22.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.911+0000 7f23a08b4640 1 --2- 172.21.15.129:0/1941020390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06c410 0x7f239c15d900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f238c0097a0 tx=0x7f238c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a540f7a00383383 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:22.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f2393fff640 1 -- 172.21.15.129:0/1941020390 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c030610 con 0x7f239c06c410 2024-03-23T15:50:22.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f2393fff640 1 -- 172.21.15.129:0/1941020390 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f238c030770 con 0x7f239c06c410 2024-03-23T15:50:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f2393fff640 1 -- 172.21.15.129:0/1941020390 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c030b00 con 0x7f239c06c410 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 -- 172.21.15.129:0/1941020390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06c410 msgr2=0x7f239c15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 --2- 172.21.15.129:0/1941020390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06c410 0x7f239c15d900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f238c0097a0 tx=0x7f238c02b960 comp rx=0 tx=0).stop 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 -- 172.21.15.129:0/1941020390 shutdown_connections 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 --2- 172.21.15.129:0/1941020390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c06c410 0x7f239c15d900 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 -- 172.21.15.129:0/1941020390 >> 172.21.15.129:0/1941020390 conn(0x7f239c0cfb20 msgr2=0x7f239c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 -- 172.21.15.129:0/1941020390 shutdown_connections 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.915+0000 7f23a3319640 1 -- 172.21.15.129:0/1941020390 wait complete. 2024-03-23T15:50:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 Processor -- start 2024-03-23T15:50:22.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 -- start start 2024-03-23T15:50:22.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f239c06c030 0x7f239c15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f239c06c410 0x7f239c1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 0x7f239c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f239c14f6e0 con 0x7f239c15de40 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.917+0000 7f23a3319640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f239c1f4210 con 0x7f239c06c030 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.918+0000 7f23a3319640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f239c1f44f0 con 0x7f239c06c410 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.918+0000 7f23a18b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 0x7f239c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.918+0000 7f23a08b4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f239c06c410 0x7f239c1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:22.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.918+0000 7f23a10b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f239c06c030 0x7f239c15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:22.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.918+0000 7f23a18b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 0x7f239c155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46464/0 (socket says 172.21.15.129:46464) 2024-03-23T15:50:22.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.918+0000 7f23a18b6640 1 -- 172.21.15.129:0/4045519456 learned_addr learned my addr 172.21.15.129:0/4045519456 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:22.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.922+0000 7f23a18b6640 1 -- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f239c06c410 msgr2=0x7f239c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:22.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.922+0000 7f23a18b6640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f239c06c410 0x7f239c1551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:22.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.922+0000 7f23a18b6640 1 -- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f239c06c030 msgr2=0x7f239c15a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:22.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.922+0000 7f23a18b6640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f239c06c030 0x7f239c15a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:22.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.922+0000 7f23a18b6640 1 -- 172.21.15.129:0/4045519456 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c009400 con 0x7f239c15de40 2024-03-23T15:50:22.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.925+0000 7f23a18b6640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 0x7f239c155ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f239400c7a0 tx=0x7f239400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:22.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.927+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2394003b70 con 0x7f239c15de40 2024-03-23T15:50:22.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.927+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239c14edb0 con 0x7f239c15de40 2024-03-23T15:50:22.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.928+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239c14f370 con 0x7f239c15de40 2024-03-23T15:50:22.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:22.929+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239c157e50 con 0x7f239c15de40 2024-03-23T15:50:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.135+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2394001830 con 0x7f239c15de40 2024-03-23T15:50:23.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.217+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239400a850 con 0x7f239c15de40 2024-03-23T15:50:23.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.218+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f239400aa70 con 0x7f239c15de40 2024-03-23T15:50:23.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.218+0000 7f2391ffb640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f238006b810 0x7f238006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:23.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.218+0000 7f23a10b5640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f238006b810 0x7f238006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:23.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.219+0000 7f23a10b5640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f238006b810 0x7f238006dcd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f23980011e0 tx=0x7f2398007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:23.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.233+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2394059070 con 0x7f239c15de40 2024-03-23T15:50:23.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.233+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2394055310 con 0x7f239c15de40 2024-03-23T15:50:23.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.512+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f239c157e50 con 0x7f239c15de40 2024-03-23T15:50:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.533+0000 7f2391ffb640 1 -- 172.21.15.129:0/4045519456 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f239c157e50 con 0x7f239c15de40 2024-03-23T15:50:23.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:23.535 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:23.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f238006b810 msgr2=0x7f238006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:23.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f238006b810 0x7f238006dcd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f23980011e0 tx=0x7f2398007d30 comp rx=0 tx=0).stop 2024-03-23T15:50:23.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 msgr2=0x7f239c155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:23.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 0x7f239c155ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f239400c7a0 tx=0x7f239400cc60 comp rx=0 tx=0).stop 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 shutdown_connections 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f239c15de40 0x7f239c155ca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f239c06c410 0x7f239c1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f238006b810 0x7f238006dcd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 --2- 172.21.15.129:0/4045519456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f239c06c030 0x7f239c15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 >> 172.21.15.129:0/4045519456 conn(0x7f239c0cfb20 msgr2=0x7f239c0ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 shutdown_connections 2024-03-23T15:50:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:23.535+0000 7f23a3319640 1 -- 172.21.15.129:0/4045519456 wait complete. 2024-03-23T15:50:24.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:24.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:25.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:25.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.256+0000 7f1fbe5c7640 1 Processor -- start 2024-03-23T15:50:25.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.256+0000 7f1fbe5c7640 1 -- start start 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fbe5c7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1fb81544b0 0x7f1fb815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fbe5c7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1fb80caa00 0x7f1fb806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fbe5c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb806c0b0 0x7f1fb80c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fbe5c7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f1fb80c5be0 con 0x7f1fb80caa00 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fbe5c7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1fb80cbcb0 con 0x7f1fb81544b0 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fbe5c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1fb815b280 con 0x7f1fb806c0b0 2024-03-23T15:50:25.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.257+0000 7f1fb77fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1fb80caa00 0x7f1fb806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:25.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.258+0000 7f1fb7fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1fb81544b0 0x7f1fb815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:25.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.258+0000 7f1fbcb64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb806c0b0 0x7f1fb80c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:25.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.258+0000 7f1fb77fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1fb80caa00 0x7f1fb806bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46470/0 (socket says 172.21.15.129:46470) 2024-03-23T15:50:25.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.258+0000 7f1fb77fe640 1 -- 172.21.15.129:0/2849793526 learned_addr learned my addr 172.21.15.129:0/2849793526 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:25.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.260+0000 7f1fb77fe640 1 -- 172.21.15.129:0/2849793526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb806c0b0 msgr2=0x7f1fb80c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:25.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.260+0000 7f1fb77fe640 1 --2- 172.21.15.129:0/2849793526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb806c0b0 0x7f1fb80c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.260+0000 7f1fb77fe640 1 -- 172.21.15.129:0/2849793526 >> v2:172.21.15.157:3300/0 conn(0x7f1fb81544b0 msgr2=0x7f1fb815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:25.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.260+0000 7f1fb77fe640 1 --2- 172.21.15.129:0/2849793526 >> v2:172.21.15.157:3300/0 conn(0x7f1fb81544b0 0x7f1fb815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.260+0000 7f1fb77fe640 1 -- 172.21.15.129:0/2849793526 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb815a9e0 con 0x7f1fb80caa00 2024-03-23T15:50:25.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.262+0000 7f1fb77fe640 1 --2- 172.21.15.129:0/2849793526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb806bb70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1fb806cd70 tx=0x7f1fac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228c08d82501bd2c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:25.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fb6ffd640 1 -- 172.21.15.129:0/2849793526 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac030860 con 0x7f1fb80caa00 2024-03-23T15:50:25.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fb6ffd640 1 -- 172.21.15.129:0/2849793526 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fac0309c0 con 0x7f1fb80caa00 2024-03-23T15:50:25.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fb6ffd640 1 -- 172.21.15.129:0/2849793526 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac030d50 con 0x7f1fb80caa00 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/2849793526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 msgr2=0x7f1fb806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/2849793526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb806bb70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1fb806cd70 tx=0x7f1fac02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/2849793526 shutdown_connections 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/2849793526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb806c0b0 0x7f1fb80c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/2849793526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb806bb70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/2849793526 >> v2:172.21.15.157:3300/0 conn(0x7f1fb81544b0 0x7f1fb815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.266+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/2849793526 >> 172.21.15.129:0/2849793526 conn(0x7f1fb8061b50 msgr2=0x7f1fb8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.267+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/2849793526 shutdown_connections 2024-03-23T15:50:25.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.267+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/2849793526 wait complete. 2024-03-23T15:50:25.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.268+0000 7f1fbe5c7640 1 Processor -- start 2024-03-23T15:50:25.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.268+0000 7f1fbe5c7640 1 -- start start 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbe5c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fb806c0b0 0x7f1fb814b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbe5c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb814a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbe5c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb81544b0 0x7f1fb8156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbe5c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb80d0630 con 0x7f1fb80caa00 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbe5c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb814ba70 con 0x7f1fb806c0b0 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbe5c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1fb8157b20 con 0x7f1fb81544b0 2024-03-23T15:50:25.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fb77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb814a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:25.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fbcb64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb81544b0 0x7f1fb8156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:25.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fb77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb814a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46478/0 (socket says 172.21.15.129:46478) 2024-03-23T15:50:25.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.269+0000 7f1fb77fe640 1 -- 172.21.15.129:0/1769688215 learned_addr learned my addr 172.21.15.129:0/1769688215 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:25.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.273+0000 7f1fb77fe640 1 -- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb81544b0 msgr2=0x7f1fb8156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:25.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.273+0000 7f1fb77fe640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb81544b0 0x7f1fb8156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.273+0000 7f1fb77fe640 1 -- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fb806c0b0 msgr2=0x7f1fb814b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:25.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.273+0000 7f1fb77fe640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fb806c0b0 0x7f1fb814b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.273+0000 7f1fb77fe640 1 -- 172.21.15.129:0/1769688215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac0094d0 con 0x7f1fb80caa00 2024-03-23T15:50:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.275+0000 7f1fb77fe640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb814a740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1fac02b9e0 tx=0x7f1fac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:25.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.280+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac039c90 con 0x7f1fb80caa00 2024-03-23T15:50:25.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.280+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fac039e10 con 0x7f1fb80caa00 2024-03-23T15:50:25.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.280+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb80cfab0 con 0x7f1fb80caa00 2024-03-23T15:50:25.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.280+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac043640 con 0x7f1fb80caa00 2024-03-23T15:50:25.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.280+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb80d0070 con 0x7f1fb80caa00 2024-03-23T15:50:25.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.281+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb80b3450 con 0x7f1fb80caa00 2024-03-23T15:50:25.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.304+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1fac0437a0 con 0x7f1fb80caa00 2024-03-23T15:50:25.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.305+0000 7f1fb4ff9640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fa406b840 0x7f1fa406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:25.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.305+0000 7f1fb7fff640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fa406b840 0x7f1fa406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:25.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.306+0000 7f1fb7fff640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fa406b840 0x7f1fa406dd00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8006a90 tx=0x7f1fa8006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:25.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.575+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1fac03e070 con 0x7f1fb80caa00 2024-03-23T15:50:25.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.575+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1fac078620 con 0x7f1fb80caa00 2024-03-23T15:50:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.800+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fb8075740 con 0x7f1fb80caa00 2024-03-23T15:50:25.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.826+0000 7f1fb4ff9640 1 -- 172.21.15.129:0/1769688215 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1fac034070 con 0x7f1fb80caa00 2024-03-23T15:50:25.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:25.829 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:25.830 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:25.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fa406b840 msgr2=0x7f1fa406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:25.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fa406b840 0x7f1fa406dd00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8006a90 tx=0x7f1fa8006a20 comp rx=0 tx=0).stop 2024-03-23T15:50:25.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 msgr2=0x7f1fb814a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:25.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb814a740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1fac02b9e0 tx=0x7f1fac004730 comp rx=0 tx=0).stop 2024-03-23T15:50:25.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 shutdown_connections 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fb81544b0 0x7f1fb8156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fb80caa00 0x7f1fb814a740 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fa406b840 0x7f1fa406dd00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 --2- 172.21.15.129:0/1769688215 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fb806c0b0 0x7f1fb814b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 >> 172.21.15.129:0/1769688215 conn(0x7f1fb8061b50 msgr2=0x7f1fb80b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 shutdown_connections 2024-03-23T15:50:25.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:25.828+0000 7f1fbe5c7640 1 -- 172.21.15.129:0/1769688215 wait complete. 2024-03-23T15:50:26.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:26.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:27.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:27.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.546+0000 7fe65fc47640 1 Processor -- start 2024-03-23T15:50:27.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.547+0000 7fe65fc47640 1 -- start start 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.547+0000 7fe65fc47640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe65815c000 0x7fe65815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.547+0000 7fe65fc47640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe65815b290 con 0x7fe6580ccd60 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.547+0000 7fe65fc47640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6580ce800 con 0x7fe6580ceb00 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.547+0000 7fe65fc47640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe658154a70 con 0x7fe65815c000 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.548+0000 7fe65e1e4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe65815c000 0x7fe65815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.548+0000 7fe65d1e2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe6580ceb00 0x7fe65815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33284/0 (socket says 172.21.15.129:33284) 2024-03-23T15:50:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.548+0000 7fe65d1e2640 1 -- 172.21.15.129:0/4279723373 learned_addr learned my addr 172.21.15.129:0/4279723373 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:27.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.551+0000 7fe65e1e4640 1 -- 172.21.15.129:0/4279723373 >> v1:172.21.15.157:6789/0 conn(0x7fe6580ceb00 legacy=0x7fe65815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:27.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.551+0000 7fe65e1e4640 1 -- 172.21.15.129:0/4279723373 >> v1:172.21.15.129:6789/0 conn(0x7fe6580ccd60 legacy=0x7fe65806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:27.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.551+0000 7fe65e1e4640 1 -- 172.21.15.129:0/4279723373 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6580cd950 con 0x7fe65815c000 2024-03-23T15:50:27.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.554+0000 7fe65e1e4640 1 --2- 172.21.15.129:0/4279723373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe65815c000 0x7fe65815ad50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe650009d90 tx=0x7fe65002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca2278bc11e7eae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.559+0000 7fe65c9e1640 1 -- 172.21.15.129:0/4279723373 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65002cd00 con 0x7fe65815c000 2024-03-23T15:50:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.559+0000 7fe65c9e1640 1 -- 172.21.15.129:0/4279723373 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe65002ce60 con 0x7fe65815c000 2024-03-23T15:50:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.559+0000 7fe65c9e1640 1 -- 172.21.15.129:0/4279723373 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe650034660 con 0x7fe65815c000 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 -- 172.21.15.129:0/4279723373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe65815c000 msgr2=0x7fe65815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4279723373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe65815c000 0x7fe65815ad50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe650009d90 tx=0x7fe65002bf50 comp rx=0 tx=0).stop 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 -- 172.21.15.129:0/4279723373 shutdown_connections 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4279723373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe65815c000 0x7fe65815ad50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 -- 172.21.15.129:0/4279723373 >> 172.21.15.129:0/4279723373 conn(0x7fe6580b1840 msgr2=0x7fe6580b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 -- 172.21.15.129:0/4279723373 shutdown_connections 2024-03-23T15:50:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.560+0000 7fe65fc47640 1 -- 172.21.15.129:0/4279723373 wait complete. 2024-03-23T15:50:27.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.562+0000 7fe65fc47640 1 Processor -- start 2024-03-23T15:50:27.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.562+0000 7fe65fc47640 1 -- start start 2024-03-23T15:50:27.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.563+0000 7fe65fc47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6580ccd60 0x7fe658153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:27.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.563+0000 7fe65fc47640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6580ceb00 0x7fe65814dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.563+0000 7fe65fc47640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe65815c000 0x7fe6581e5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.563+0000 7fe65fc47640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe658158d20 con 0x7fe6580ccd60 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.563+0000 7fe65fc47640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe65815d550 con 0x7fe6580ceb00 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.563+0000 7fe65fc47640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe6581f3d90 con 0x7fe65815c000 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.564+0000 7fe65e1e4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe65815c000 0x7fe6581e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.564+0000 7fe65d1e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6580ceb00 0x7fe65814dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.564+0000 7fe65d9e3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6580ccd60 0x7fe658153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:27.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.564+0000 7fe65e1e4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe65815c000 0x7fe6581e5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:59232/0 (socket says 172.21.15.129:59232) 2024-03-23T15:50:27.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.564+0000 7fe65e1e4640 1 -- 172.21.15.129:0/4068329184 learned_addr learned my addr 172.21.15.129:0/4068329184 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.569+0000 7fe65d9e3640 1 -- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe65815c000 msgr2=0x7fe6581e5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.569+0000 7fe65d9e3640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe65815c000 0x7fe6581e5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.569+0000 7fe65d9e3640 1 -- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6580ceb00 msgr2=0x7fe65814dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.569+0000 7fe65d9e3640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6580ceb00 0x7fe65814dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.569+0000 7fe65d9e3640 1 -- 172.21.15.129:0/4068329184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe650009a70 con 0x7fe6580ccd60 2024-03-23T15:50:27.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.571+0000 7fe65d9e3640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6580ccd60 0x7fe658153a20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe65400b410 tx=0x7fe65400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:27.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.576+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654004020 con 0x7fe6580ccd60 2024-03-23T15:50:27.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.576+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe654004180 con 0x7fe6580ccd60 2024-03-23T15:50:27.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.576+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654011630 con 0x7fe6580ccd60 2024-03-23T15:50:27.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.576+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6581f4050 con 0x7fe6580ccd60 2024-03-23T15:50:27.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.576+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6581f4610 con 0x7fe6580ccd60 2024-03-23T15:50:27.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.577+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65804f530 con 0x7fe6580ccd60 2024-03-23T15:50:27.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.831+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe654011790 con 0x7fe6580ccd60 2024-03-23T15:50:27.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.832+0000 7fe64e7fc640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe63c06b890 0x7fe63c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:27.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.832+0000 7fe65d1e2640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe63c06b890 0x7fe63c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:27.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.833+0000 7fe65d1e2640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe63c06b890 0x7fe63c06dd50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe658158710 tx=0x7fe648007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:27.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.846+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe654058d70 con 0x7fe6580ccd60 2024-03-23T15:50:27.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:27.847+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe654055010 con 0x7fe6580ccd60 2024-03-23T15:50:28.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.089+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6580686f0 con 0x7fe6580ccd60 2024-03-23T15:50:28.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.101+0000 7fe64e7fc640 1 -- 172.21.15.129:0/4068329184 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe654015090 con 0x7fe6580ccd60 2024-03-23T15:50:28.104 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:28.104 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:28.104 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe63c06b890 msgr2=0x7fe63c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe63c06b890 0x7fe63c06dd50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe658158710 tx=0x7fe648007e80 comp rx=0 tx=0).stop 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6580ccd60 msgr2=0x7fe658153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6580ccd60 0x7fe658153a20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe65400b410 tx=0x7fe65400b8d0 comp rx=0 tx=0).stop 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 shutdown_connections 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe65815c000 0x7fe6581e5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe63c06b890 0x7fe63c06dd50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6580ceb00 0x7fe65814dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 --2- 172.21.15.129:0/4068329184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6580ccd60 0x7fe658153a20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:28.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 >> 172.21.15.129:0/4068329184 conn(0x7fe6580b1840 msgr2=0x7fe65814d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:28.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 shutdown_connections 2024-03-23T15:50:28.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:28.103+0000 7fe65fc47640 1 -- 172.21.15.129:0/4068329184 wait complete. 2024-03-23T15:50:28.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:28.722 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:29.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:29.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.822+0000 7fc90d702640 1 Processor -- start 2024-03-23T15:50:29.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.822+0000 7fc90d702640 1 -- start start 2024-03-23T15:50:29.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.823+0000 7fc90d702640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc90815a2e0 0x7fc90806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:29.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.823+0000 7fc90d702640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9080cbcb0 con 0x7fc90806c0b0 2024-03-23T15:50:29.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.823+0000 7fc90d702640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc90815b280 con 0x7fc9081544b0 2024-03-23T15:50:29.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.823+0000 7fc90d702640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc9080c5be0 con 0x7fc90815a2e0 2024-03-23T15:50:29.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.824+0000 7fc9077fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc90806c0b0 0x7fc9080c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:41588/0 (socket says 172.21.15.129:41588) 2024-03-23T15:50:29.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.824+0000 7fc9077fe640 1 -- 172.21.15.129:0/3495077583 learned_addr learned my addr 172.21.15.129:0/3495077583 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:29.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.824+0000 7fc9067fc640 1 --2- 172.21.15.129:0/3495077583 >> v2:172.21.15.157:3300/0 conn(0x7fc90815a2e0 0x7fc90806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:29.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.826+0000 7fc905ffb640 1 -- 172.21.15.129:0/3495077583 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761247544 0 0) 0x7fc9080cbcb0 con 0x7fc90806c0b0 2024-03-23T15:50:29.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.826+0000 7fc905ffb640 1 -- 172.21.15.129:0/3495077583 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8f0003520 con 0x7fc90806c0b0 2024-03-23T15:50:29.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.830+0000 7fc9067fc640 1 -- 172.21.15.129:0/3495077583 >> v1:172.21.15.157:6789/0 conn(0x7fc9081544b0 legacy=0x7fc9080caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:29.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.830+0000 7fc9067fc640 1 -- 172.21.15.129:0/3495077583 >> v1:172.21.15.129:6789/0 conn(0x7fc90806c0b0 legacy=0x7fc9080c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:29.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.830+0000 7fc9067fc640 1 -- 172.21.15.129:0/3495077583 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90815a9e0 con 0x7fc90815a2e0 2024-03-23T15:50:29.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.837+0000 7fc9067fc640 1 --2- 172.21.15.129:0/3495077583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90815a2e0 0x7fc90806bb70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0097a0 tx=0x7fc8fc02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fb13fc33cdf964d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.843+0000 7fc905ffb640 1 -- 172.21.15.129:0/3495077583 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc02f810 con 0x7fc90815a2e0 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.843+0000 7fc905ffb640 1 -- 172.21.15.129:0/3495077583 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc02f970 con 0x7fc90815a2e0 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.843+0000 7fc905ffb640 1 -- 172.21.15.129:0/3495077583 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc02fd00 con 0x7fc90815a2e0 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.843+0000 7fc90d702640 1 -- 172.21.15.129:0/3495077583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90815a2e0 msgr2=0x7fc90806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.844+0000 7fc90d702640 1 --2- 172.21.15.129:0/3495077583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90815a2e0 0x7fc90806bb70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0097a0 tx=0x7fc8fc02b9f0 comp rx=0 tx=0).stop 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.844+0000 7fc90d702640 1 -- 172.21.15.129:0/3495077583 shutdown_connections 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.844+0000 7fc90d702640 1 --2- 172.21.15.129:0/3495077583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90815a2e0 0x7fc90806bb70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.844+0000 7fc90d702640 1 -- 172.21.15.129:0/3495077583 >> 172.21.15.129:0/3495077583 conn(0x7fc908061b50 msgr2=0x7fc908061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.844+0000 7fc90d702640 1 -- 172.21.15.129:0/3495077583 shutdown_connections 2024-03-23T15:50:29.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.844+0000 7fc90d702640 1 -- 172.21.15.129:0/3495077583 wait complete. 2024-03-23T15:50:29.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.846+0000 7fc90d702640 1 Processor -- start 2024-03-23T15:50:29.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.846+0000 7fc90d702640 1 -- start start 2024-03-23T15:50:29.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc90d702640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90806c0b0 0x7fc9080b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:29.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc90d702640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9081544b0 0x7fc908155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:29.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc90d702640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90815a2e0 0x7fc908155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:29.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc90d702640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc908157a10 con 0x7fc90815a2e0 2024-03-23T15:50:29.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc90d702640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9081f3b50 con 0x7fc9081544b0 2024-03-23T15:50:29.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc90d702640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9081f3870 con 0x7fc90806c0b0 2024-03-23T15:50:29.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.847+0000 7fc9077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90815a2e0 0x7fc908155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:29.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.848+0000 7fc9067fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9081544b0 0x7fc908155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:29.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.848+0000 7fc906ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90806c0b0 0x7fc9080b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:29.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.848+0000 7fc9077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90815a2e0 0x7fc908155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46518/0 (socket says 172.21.15.129:46518) 2024-03-23T15:50:29.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.848+0000 7fc9077fe640 1 -- 172.21.15.129:0/2002608015 learned_addr learned my addr 172.21.15.129:0/2002608015 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:29.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.851+0000 7fc906ffd640 1 -- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9081544b0 msgr2=0x7fc908155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:29.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.851+0000 7fc906ffd640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9081544b0 0x7fc908155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:29.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.851+0000 7fc906ffd640 1 -- 172.21.15.129:0/2002608015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90815a2e0 msgr2=0x7fc908155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:29.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.851+0000 7fc906ffd640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90815a2e0 0x7fc908155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:29.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.851+0000 7fc906ffd640 1 -- 172.21.15.129:0/2002608015 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc009400 con 0x7fc90806c0b0 2024-03-23T15:50:29.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.854+0000 7fc906ffd640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90806c0b0 0x7fc9080b4160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800c390 tx=0x7fc8f800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:29.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.858+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80052d0 con 0x7fc90806c0b0 2024-03-23T15:50:29.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.858+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8f8005430 con 0x7fc90806c0b0 2024-03-23T15:50:29.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.858+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8003830 con 0x7fc90806c0b0 2024-03-23T15:50:29.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.858+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc908154c70 con 0x7fc90806c0b0 2024-03-23T15:50:29.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.858+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc908155230 con 0x7fc90806c0b0 2024-03-23T15:50:29.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:29.860+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc908075740 con 0x7fc90806c0b0 2024-03-23T15:50:30.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.145+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc8f8003990 con 0x7fc90806c0b0 2024-03-23T15:50:30.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.145+0000 7fc8ef7fe640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc8f406b860 0x7fc8f406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:30.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.146+0000 7fc9067fc640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc8f406b860 0x7fc8f406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:30.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.146+0000 7fc9067fc640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc8f406b860 0x7fc8f406dd20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc036000 tx=0x7fc8fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:30.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.152+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc8f8058380 con 0x7fc90806c0b0 2024-03-23T15:50:30.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.166+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc8f8054620 con 0x7fc90806c0b0 2024-03-23T15:50:30.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.383+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9080b4bb0 con 0x7fc90806c0b0 2024-03-23T15:50:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.397+0000 7fc8ef7fe640 1 -- 172.21.15.129:0/2002608015 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc8f8017090 con 0x7fc90806c0b0 2024-03-23T15:50:30.400 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:30.400 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc8f406b860 msgr2=0x7fc8f406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc8f406b860 0x7fc8f406dd20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc036000 tx=0x7fc8fc008030 comp rx=0 tx=0).stop 2024-03-23T15:50:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90806c0b0 msgr2=0x7fc9080b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90806c0b0 0x7fc9080b4160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800c390 tx=0x7fc8f800c850 comp rx=0 tx=0).stop 2024-03-23T15:50:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 shutdown_connections 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90815a2e0 0x7fc908155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc8f406b860 0x7fc8f406dd20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9081544b0 0x7fc908155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 --2- 172.21.15.129:0/2002608015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90806c0b0 0x7fc9080b4160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 >> 172.21.15.129:0/2002608015 conn(0x7fc908061b50 msgr2=0x7fc9080c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 shutdown_connections 2024-03-23T15:50:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:30.399+0000 7fc90d702640 1 -- 172.21.15.129:0/2002608015 wait complete. 2024-03-23T15:50:31.018 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:31.019 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:32.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:32.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.127+0000 7f51535f4640 1 Processor -- start 2024-03-23T15:50:32.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.127+0000 7f51535f4640 1 -- start start 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.128+0000 7f51535f4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f514c0ceb00 0x7f514c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.128+0000 7f51535f4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f514c0cef40 0x7f514c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.128+0000 7f51535f4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514c15b290 con 0x7f514c0ccd60 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.128+0000 7f51535f4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f514c0ce800 con 0x7f514c0cef40 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.128+0000 7f51535f4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f514c154a70 con 0x7f514c0ceb00 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.129+0000 7f5150b8f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f514c0ceb00 0x7f514c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:32.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.129+0000 7f5151b91640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f514c0cef40 0x7f514c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:32.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.129+0000 7f5150b8f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f514c0ceb00 0x7f514c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43372/0 (socket says 172.21.15.129:43372) 2024-03-23T15:50:32.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.129+0000 7f5150b8f640 1 -- 172.21.15.129:0/2149467513 learned_addr learned my addr 172.21.15.129:0/2149467513 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:32.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.132+0000 7f5150b8f640 1 -- 172.21.15.129:0/2149467513 >> v2:172.21.15.129:3300/0 conn(0x7f514c0cef40 msgr2=0x7f514c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:32.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.132+0000 7f5150b8f640 1 --2- 172.21.15.129:0/2149467513 >> v2:172.21.15.129:3300/0 conn(0x7f514c0cef40 0x7f514c15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.132+0000 7f5150b8f640 1 -- 172.21.15.129:0/2149467513 >> v1:172.21.15.157:6789/0 conn(0x7f514c0ccd60 legacy=0x7f514c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:32.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.132+0000 7f5150b8f640 1 -- 172.21.15.129:0/2149467513 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c0cd950 con 0x7f514c0ceb00 2024-03-23T15:50:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.135+0000 7f5150b8f640 1 --2- 172.21.15.129:0/2149467513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0ceb00 0x7f514c15ad50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f514c08d1a0 tx=0x7f513c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7745b8a4a7a59149 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:32.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.141+0000 7f5143fff640 1 -- 172.21.15.129:0/2149467513 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c030860 con 0x7f514c0ceb00 2024-03-23T15:50:32.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.141+0000 7f5143fff640 1 -- 172.21.15.129:0/2149467513 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f513c0309c0 con 0x7f514c0ceb00 2024-03-23T15:50:32.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.141+0000 7f5143fff640 1 -- 172.21.15.129:0/2149467513 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c030d50 con 0x7f514c0ceb00 2024-03-23T15:50:32.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 -- 172.21.15.129:0/2149467513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0ceb00 msgr2=0x7f514c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:32.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 --2- 172.21.15.129:0/2149467513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0ceb00 0x7f514c15ad50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f514c08d1a0 tx=0x7f513c02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 -- 172.21.15.129:0/2149467513 shutdown_connections 2024-03-23T15:50:32.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 --2- 172.21.15.129:0/2149467513 >> v2:172.21.15.129:3300/0 conn(0x7f514c0cef40 0x7f514c15b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 --2- 172.21.15.129:0/2149467513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0ceb00 0x7f514c15ad50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 -- 172.21.15.129:0/2149467513 >> 172.21.15.129:0/2149467513 conn(0x7f514c0b1840 msgr2=0x7f514c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:32.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 -- 172.21.15.129:0/2149467513 shutdown_connections 2024-03-23T15:50:32.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.142+0000 7f51535f4640 1 -- 172.21.15.129:0/2149467513 wait complete. 2024-03-23T15:50:32.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.143+0000 7f51535f4640 1 Processor -- start 2024-03-23T15:50:32.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.143+0000 7f51535f4640 1 -- start start 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.144+0000 7f51535f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 0x7f514c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.144+0000 7f51535f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f514c0ceb00 0x7f514c1e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.144+0000 7f51535f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0cef40 0x7f514c1e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.144+0000 7f51535f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f514c0cc2b0 con 0x7f514c0ceb00 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.144+0000 7f51535f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f514c0c7fc0 con 0x7f514c0cef40 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.144+0000 7f51535f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f514c0cbff0 con 0x7f514c0ccd60 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.145+0000 7f5151b91640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0cef40 0x7f514c1e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.145+0000 7f5150b8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f514c0ceb00 0x7f514c1e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.145+0000 7f5151390640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 0x7f514c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.145+0000 7f5151390640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 0x7f514c0c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51300/0 (socket says 172.21.15.129:51300) 2024-03-23T15:50:32.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.145+0000 7f5151390640 1 -- 172.21.15.129:0/3095180659 learned_addr learned my addr 172.21.15.129:0/3095180659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:32.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.149+0000 7f5151390640 1 -- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0cef40 msgr2=0x7f514c1e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:32.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.149+0000 7f5151390640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0cef40 0x7f514c1e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.149+0000 7f5151390640 1 -- 172.21.15.129:0/3095180659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f514c0ceb00 msgr2=0x7f514c1e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:32.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.149+0000 7f5151390640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f514c0ceb00 0x7f514c1e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.149+0000 7f5151390640 1 -- 172.21.15.129:0/3095180659 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0094d0 con 0x7f514c0ccd60 2024-03-23T15:50:32.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.151+0000 7f5151390640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 0x7f514c0c6dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f514800b3e0 tx=0x7f514800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:32.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.155+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148011830 con 0x7f514c0ccd60 2024-03-23T15:50:32.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.155+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51480119b0 con 0x7f514c0ccd60 2024-03-23T15:50:32.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.155+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c1f0d60 con 0x7f514c0ccd60 2024-03-23T15:50:32.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.155+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148011d60 con 0x7f514c0ccd60 2024-03-23T15:50:32.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.155+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c1e36e0 con 0x7f514c0ccd60 2024-03-23T15:50:32.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.156+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514c04f530 con 0x7f514c0ccd60 2024-03-23T15:50:32.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.432+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f514801a400 con 0x7f514c0ccd60 2024-03-23T15:50:32.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.433+0000 7f5141ffb640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f513006b810 0x7f513006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:32.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.433+0000 7f5150b8f640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f513006b810 0x7f513006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:32.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.433+0000 7f5150b8f640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f513006b810 0x7f513006dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f513c036000 tx=0x7f513c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:32.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.436+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5148059070 con 0x7f514c0ccd60 2024-03-23T15:50:32.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.449+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5148055310 con 0x7f514c0ccd60 2024-03-23T15:50:32.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.681+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f514c06db50 con 0x7f514c0ccd60 2024-03-23T15:50:32.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.752+0000 7f5141ffb640 1 -- 172.21.15.129:0/3095180659 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5148015020 con 0x7f514c0ccd60 2024-03-23T15:50:32.756 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:32.756 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:32.756 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:32.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f513006b810 msgr2=0x7f513006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f513006b810 0x7f513006dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f513c036000 tx=0x7f513c008030 comp rx=0 tx=0).stop 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 msgr2=0x7f514c0c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 0x7f514c0c6dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f514800b3e0 tx=0x7f514800b8a0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 shutdown_connections 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f513006b810 0x7f513006dcd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f514c0cef40 0x7f514c1e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f514c0ceb00 0x7f514c1e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 --2- 172.21.15.129:0/3095180659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f514c0ccd60 0x7f514c0c6dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 >> 172.21.15.129:0/3095180659 conn(0x7f514c0b1840 msgr2=0x7f514c1599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 shutdown_connections 2024-03-23T15:50:32.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:32.754+0000 7f51535f4640 1 -- 172.21.15.129:0/3095180659 wait complete. 2024-03-23T15:50:33.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:33.372 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:34.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:34.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.474+0000 7f09cce84640 1 Processor -- start 2024-03-23T15:50:34.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.474+0000 7f09cce84640 1 -- start start 2024-03-23T15:50:34.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09cce84640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f09c815c000 0x7f09c815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:34.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09cce84640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c815b290 con 0x7f09c80ceb00 2024-03-23T15:50:34.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09cce84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c80ce800 con 0x7f09c80ccd60 2024-03-23T15:50:34.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09cce84640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f09c8154a70 con 0x7f09c815c000 2024-03-23T15:50:34.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09c6d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f09c815c000 0x7f09c815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:34.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09c5d9b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f09c80ceb00 0x7f09c815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:39300/0 (socket says 172.21.15.129:39300) 2024-03-23T15:50:34.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.475+0000 7f09c5d9b640 1 -- 172.21.15.129:0/1519090322 learned_addr learned my addr 172.21.15.129:0/1519090322 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:34.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.478+0000 7f09c6d9d640 1 -- 172.21.15.129:0/1519090322 >> v1:172.21.15.157:6789/0 conn(0x7f09c80ccd60 legacy=0x7f09c806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:34.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.478+0000 7f09c6d9d640 1 -- 172.21.15.129:0/1519090322 >> v1:172.21.15.129:6789/0 conn(0x7f09c80ceb00 legacy=0x7f09c815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:34.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.478+0000 7f09c6d9d640 1 -- 172.21.15.129:0/1519090322 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c80cd950 con 0x7f09c815c000 2024-03-23T15:50:34.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.480+0000 7f09c6d9d640 1 --2- 172.21.15.129:0/1519090322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c815ad50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009d90 tx=0x7f09c002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f092693c7639654 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:34.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09c559a640 1 -- 172.21.15.129:0/1519090322 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c002cd00 con 0x7f09c815c000 2024-03-23T15:50:34.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09c559a640 1 -- 172.21.15.129:0/1519090322 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c002ce60 con 0x7f09c815c000 2024-03-23T15:50:34.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09c559a640 1 -- 172.21.15.129:0/1519090322 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0034660 con 0x7f09c815c000 2024-03-23T15:50:34.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09cce84640 1 -- 172.21.15.129:0/1519090322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 msgr2=0x7f09c815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:34.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09cce84640 1 --2- 172.21.15.129:0/1519090322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c815ad50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009d90 tx=0x7f09c002bf50 comp rx=0 tx=0).stop 2024-03-23T15:50:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09cce84640 1 -- 172.21.15.129:0/1519090322 shutdown_connections 2024-03-23T15:50:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.485+0000 7f09cce84640 1 --2- 172.21.15.129:0/1519090322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c815ad50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.486+0000 7f09cce84640 1 -- 172.21.15.129:0/1519090322 >> 172.21.15.129:0/1519090322 conn(0x7f09c80b1840 msgr2=0x7f09c80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.486+0000 7f09cce84640 1 -- 172.21.15.129:0/1519090322 shutdown_connections 2024-03-23T15:50:34.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.486+0000 7f09cce84640 1 -- 172.21.15.129:0/1519090322 wait complete. 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.487+0000 7f09cce84640 1 Processor -- start 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.487+0000 7f09cce84640 1 -- start start 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09cce84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c80ccd60 0x7f09c81f0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09cce84640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 0x7f09c81f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09cce84640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c81e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09cce84640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09c80c9560 con 0x7f09c80ccd60 2024-03-23T15:50:34.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09cce84640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f09c81f13b0 con 0x7f09c815c000 2024-03-23T15:50:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09cce84640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f09c8159ad0 con 0x7f09c80ceb00 2024-03-23T15:50:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09c659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c80ccd60 0x7f09c81f0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09c5d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 0x7f09c81f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.488+0000 7f09c6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c81e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.489+0000 7f09c5d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 0x7f09c81f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51302/0 (socket says 172.21.15.129:51302) 2024-03-23T15:50:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.489+0000 7f09c5d9b640 1 -- 172.21.15.129:0/4003592070 learned_addr learned my addr 172.21.15.129:0/4003592070 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.492+0000 7f09c5d9b640 1 -- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 msgr2=0x7f09c81e5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.492+0000 7f09c5d9b640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c81e5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.492+0000 7f09c5d9b640 1 -- 172.21.15.129:0/4003592070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c80ccd60 msgr2=0x7f09c81f0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.492+0000 7f09c5d9b640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c80ccd60 0x7f09c81f0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:34.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.492+0000 7f09c5d9b640 1 -- 172.21.15.129:0/4003592070 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c0009a70 con 0x7f09c80ceb00 2024-03-23T15:50:34.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.494+0000 7f09c5d9b640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 0x7f09c81f4320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c400 tx=0x7f09bc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:34.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.497+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc005ce0 con 0x7f09c80ceb00 2024-03-23T15:50:34.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.497+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c81f3e90 con 0x7f09c80ceb00 2024-03-23T15:50:34.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.497+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c8159d20 con 0x7f09c80ceb00 2024-03-23T15:50:34.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.498+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c804f530 con 0x7f09c80ceb00 2024-03-23T15:50:34.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.499+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09bc005e40 con 0x7f09c80ceb00 2024-03-23T15:50:34.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.805+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0072d0 con 0x7f09c80ceb00 2024-03-23T15:50:34.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.806+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09bc0074f0 con 0x7f09c80ceb00 2024-03-23T15:50:34.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.806+0000 7f09b6ffd640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09a806b840 0x7f09a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:34.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.807+0000 7f09c659c640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09a806b840 0x7f09a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.807+0000 7f09c659c640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09a806b840 0x7f09a806dd00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09b8006fd0 tx=0x7f09b8009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:34.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.812+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f09bc058610 con 0x7f09c80ceb00 2024-03-23T15:50:34.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:34.830+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09bc0548b0 con 0x7f09c80ceb00 2024-03-23T15:50:35.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.152+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09c80686f0 con 0x7f09c80ceb00 2024-03-23T15:50:35.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.173+0000 7f09b6ffd640 1 -- 172.21.15.129:0/4003592070 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f09bc021020 con 0x7f09c80ceb00 2024-03-23T15:50:35.177 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:35.177 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:35.177 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09a806b840 msgr2=0x7f09a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09a806b840 0x7f09a806dd00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09b8006fd0 tx=0x7f09b8009060 comp rx=0 tx=0).stop 2024-03-23T15:50:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 msgr2=0x7f09c81f4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 0x7f09c81f4320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c400 tx=0x7f09bc00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:50:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 shutdown_connections 2024-03-23T15:50:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09a806b840 0x7f09a806dd00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09c815c000 0x7f09c81e5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09c80ceb00 0x7f09c81f4320 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 --2- 172.21.15.129:0/4003592070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09c80ccd60 0x7f09c81f0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 >> 172.21.15.129:0/4003592070 conn(0x7f09c80b1840 msgr2=0x7f09c814d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 shutdown_connections 2024-03-23T15:50:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:35.175+0000 7f09cce84640 1 -- 172.21.15.129:0/4003592070 wait complete. 2024-03-23T15:50:35.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:35.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:36.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:50:36.436 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:50:36.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:36.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.897+0000 7fd7fe363640 1 Processor -- start 2024-03-23T15:50:36.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.897+0000 7fd7fe363640 1 -- start start 2024-03-23T15:50:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.898+0000 7fd7fe363640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 0x7fd7f815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.898+0000 7fd7fe363640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd7f80ceb00 0x7fd7f806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.898+0000 7fd7fe363640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f815b290 con 0x7fd7f80cef40 2024-03-23T15:50:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.898+0000 7fd7fe363640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f80ce800 con 0x7fd7f80ceb00 2024-03-23T15:50:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.898+0000 7fd7fe363640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f8154a70 con 0x7fd7f80ccd60 2024-03-23T15:50:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.899+0000 7fd7f7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 0x7fd7f815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:36.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.900+0000 7fd7f7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 0x7fd7f815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51308/0 (socket says 172.21.15.129:51308) 2024-03-23T15:50:36.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.900+0000 7fd7f7fff640 1 -- 172.21.15.129:0/2684650652 learned_addr learned my addr 172.21.15.129:0/2684650652 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.903+0000 7fd7f7fff640 1 -- 172.21.15.129:0/2684650652 >> v2:172.21.15.129:3300/0 conn(0x7fd7f80ceb00 msgr2=0x7fd7f806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.903+0000 7fd7f7fff640 1 --2- 172.21.15.129:0/2684650652 >> v2:172.21.15.129:3300/0 conn(0x7fd7f80ceb00 0x7fd7f806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.903+0000 7fd7f7fff640 1 -- 172.21.15.129:0/2684650652 >> v1:172.21.15.157:6789/0 conn(0x7fd7f80cef40 legacy=0x7fd7f815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.903+0000 7fd7f7fff640 1 -- 172.21.15.129:0/2684650652 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f80cd950 con 0x7fd7f80ccd60 2024-03-23T15:50:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.906+0000 7fd7f7fff640 1 --2- 172.21.15.129:0/2684650652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 0x7fd7f815ad50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009870 tx=0x7fd7e802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad7f5d7153a4d677 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:36.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.910+0000 7fd7f6ffd640 1 -- 172.21.15.129:0/2684650652 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8030840 con 0x7fd7f80ccd60 2024-03-23T15:50:36.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.910+0000 7fd7f6ffd640 1 -- 172.21.15.129:0/2684650652 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e80309a0 con 0x7fd7f80ccd60 2024-03-23T15:50:36.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.910+0000 7fd7f6ffd640 1 -- 172.21.15.129:0/2684650652 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8030d30 con 0x7fd7f80ccd60 2024-03-23T15:50:36.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 -- 172.21.15.129:0/2684650652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 msgr2=0x7fd7f815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 --2- 172.21.15.129:0/2684650652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 0x7fd7f815ad50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009870 tx=0x7fd7e802b980 comp rx=0 tx=0).stop 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 -- 172.21.15.129:0/2684650652 shutdown_connections 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 --2- 172.21.15.129:0/2684650652 >> v2:172.21.15.129:3300/0 conn(0x7fd7f80ceb00 0x7fd7f806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 --2- 172.21.15.129:0/2684650652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ccd60 0x7fd7f815ad50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 -- 172.21.15.129:0/2684650652 >> 172.21.15.129:0/2684650652 conn(0x7fd7f80b1840 msgr2=0x7fd7f80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 -- 172.21.15.129:0/2684650652 shutdown_connections 2024-03-23T15:50:36.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.911+0000 7fd7fe363640 1 -- 172.21.15.129:0/2684650652 wait complete. 2024-03-23T15:50:36.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.913+0000 7fd7fe363640 1 Processor -- start 2024-03-23T15:50:36.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.913+0000 7fd7fe363640 1 -- start start 2024-03-23T15:50:36.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.914+0000 7fd7fe363640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7f80ccd60 0x7fd7f81e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:36.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.914+0000 7fd7fe363640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 0x7fd7f814f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:36.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.914+0000 7fd7fe363640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f80cef40 0x7fd7f814fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:36.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.914+0000 7fd7fe363640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8159740 con 0x7fd7f80ccd60 2024-03-23T15:50:36.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.914+0000 7fd7fe363640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f814f230 con 0x7fd7f80cef40 2024-03-23T15:50:36.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.914+0000 7fd7fe363640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f8159df0 con 0x7fd7f80ceb00 2024-03-23T15:50:36.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.915+0000 7fd7f77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 0x7fd7f814f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:36.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.916+0000 7fd7f77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 0x7fd7f814f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51316/0 (socket says 172.21.15.129:51316) 2024-03-23T15:50:36.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.916+0000 7fd7f77fe640 1 -- 172.21.15.129:0/1328755641 learned_addr learned my addr 172.21.15.129:0/1328755641 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:36.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.921+0000 7fd7f77fe640 1 -- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f80cef40 msgr2=0x7fd7f814fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:36.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.921+0000 7fd7f77fe640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f80cef40 0x7fd7f814fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:36.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.921+0000 7fd7f77fe640 1 -- 172.21.15.129:0/1328755641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7f80ccd60 msgr2=0x7fd7f81e5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:36.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.921+0000 7fd7f77fe640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7f80ccd60 0x7fd7f81e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:36.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.921+0000 7fd7f77fe640 1 -- 172.21.15.129:0/1328755641 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e80094d0 con 0x7fd7f80ceb00 2024-03-23T15:50:36.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.924+0000 7fd7f77fe640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 0x7fd7f814f710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b4a0 tx=0x7fd7ec00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:36.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.930+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec010610 con 0x7fd7f80ceb00 2024-03-23T15:50:36.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.931+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec010770 con 0x7fd7f80ceb00 2024-03-23T15:50:36.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.931+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f815a0d0 con 0x7fd7f80ceb00 2024-03-23T15:50:36.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.931+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec010b20 con 0x7fd7f80ceb00 2024-03-23T15:50:36.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.931+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f815a690 con 0x7fd7f80ceb00 2024-03-23T15:50:36.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:36.932+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f804f530 con 0x7fd7f80ceb00 2024-03-23T15:50:37.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.244+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd7ec010c80 con 0x7fd7f80ceb00 2024-03-23T15:50:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.245+0000 7fd7f4ff9640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd7e406b860 0x7fd7e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.245+0000 7fd7f7fff640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd7e406b860 0x7fd7e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:37.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.246+0000 7fd7f7fff640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd7e406b860 0x7fd7e406dd20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8036000 tx=0x7fd7e803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:37.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.249+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd7ec0590f0 con 0x7fd7f80ceb00 2024-03-23T15:50:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.266+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7ec054c90 con 0x7fd7f80ceb00 2024-03-23T15:50:37.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.517+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7f80686f0 con 0x7fd7f80ceb00 2024-03-23T15:50:37.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.531+0000 7fd7f4ff9640 1 -- 172.21.15.129:0/1328755641 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd7ec01e020 con 0x7fd7f80ceb00 2024-03-23T15:50:37.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:37.535 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:37.535 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd7e406b860 msgr2=0x7fd7e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd7e406b860 0x7fd7e406dd20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8036000 tx=0x7fd7e803b040 comp rx=0 tx=0).stop 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 msgr2=0x7fd7f814f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 0x7fd7f814f710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b4a0 tx=0x7fd7ec00b960 comp rx=0 tx=0).stop 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 shutdown_connections 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd7e406b860 0x7fd7e406dd20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f80cef40 0x7fd7f814fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f80ceb00 0x7fd7f814f710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 --2- 172.21.15.129:0/1328755641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7f80ccd60 0x7fd7f81e5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 >> 172.21.15.129:0/1328755641 conn(0x7fd7f80b1840 msgr2=0x7fd7f81e6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.533+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 shutdown_connections 2024-03-23T15:50:37.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:37.534+0000 7fd7fe363640 1 -- 172.21.15.129:0/1328755641 wait complete. 2024-03-23T15:50:38.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:38.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:39.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.257+0000 7ffa50bc7640 1 Processor -- start 2024-03-23T15:50:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.257+0000 7ffa50bc7640 1 -- start start 2024-03-23T15:50:39.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.258+0000 7ffa50bc7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffa4c1544b0 0x7ffa4c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:39.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.258+0000 7ffa50bc7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffa4c0caa00 0x7ffa4c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.258+0000 7ffa50bc7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa4c0cbcb0 con 0x7ffa4c06c0b0 2024-03-23T15:50:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.258+0000 7ffa50bc7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ffa4c0c5be0 con 0x7ffa4c0caa00 2024-03-23T15:50:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.258+0000 7ffa50bc7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffa4c15b280 con 0x7ffa4c1544b0 2024-03-23T15:50:39.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.311+0000 7ffa49d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffa4c0caa00 0x7ffa4c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:39.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.312+0000 7ffa49d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffa4c0caa00 0x7ffa4c06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50304/0 (socket says 172.21.15.129:50304) 2024-03-23T15:50:39.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.312+0000 7ffa49d9b640 1 -- 172.21.15.129:0/3736444474 learned_addr learned my addr 172.21.15.129:0/3736444474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:39.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.315+0000 7ffa49d9b640 1 -- 172.21.15.129:0/3736444474 >> v2:172.21.15.157:3300/0 conn(0x7ffa4c1544b0 msgr2=0x7ffa4c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:39.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.315+0000 7ffa49d9b640 1 --2- 172.21.15.129:0/3736444474 >> v2:172.21.15.157:3300/0 conn(0x7ffa4c1544b0 0x7ffa4c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.315+0000 7ffa49d9b640 1 -- 172.21.15.129:0/3736444474 >> v1:172.21.15.129:6789/0 conn(0x7ffa4c06c0b0 legacy=0x7ffa4c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:39.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.315+0000 7ffa49d9b640 1 -- 172.21.15.129:0/3736444474 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4c15a9e0 con 0x7ffa4c0caa00 2024-03-23T15:50:39.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.319+0000 7ffa49d9b640 1 --2- 172.21.15.129:0/3736444474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c06bb70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c06cd70 tx=0x7ffa4002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32104a7acefd3d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:39.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.323+0000 7ffa4959a640 1 -- 172.21.15.129:0/3736444474 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4002f850 con 0x7ffa4c0caa00 2024-03-23T15:50:39.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.323+0000 7ffa4959a640 1 -- 172.21.15.129:0/3736444474 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4002f9b0 con 0x7ffa4c0caa00 2024-03-23T15:50:39.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.323+0000 7ffa4959a640 1 -- 172.21.15.129:0/3736444474 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4002fd40 con 0x7ffa4c0caa00 2024-03-23T15:50:39.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 -- 172.21.15.129:0/3736444474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 msgr2=0x7ffa4c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:39.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/3736444474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c06bb70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c06cd70 tx=0x7ffa4002ba30 comp rx=0 tx=0).stop 2024-03-23T15:50:39.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 -- 172.21.15.129:0/3736444474 shutdown_connections 2024-03-23T15:50:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/3736444474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c06bb70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/3736444474 >> v2:172.21.15.157:3300/0 conn(0x7ffa4c1544b0 0x7ffa4c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 -- 172.21.15.129:0/3736444474 >> 172.21.15.129:0/3736444474 conn(0x7ffa4c061b50 msgr2=0x7ffa4c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 -- 172.21.15.129:0/3736444474 shutdown_connections 2024-03-23T15:50:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.324+0000 7ffa50bc7640 1 -- 172.21.15.129:0/3736444474 wait complete. 2024-03-23T15:50:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 Processor -- start 2024-03-23T15:50:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 -- start start 2024-03-23T15:50:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa4c06c0b0 0x7ffa4c1484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa4c1544b0 0x7ffa4c154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c157790 con 0x7ffa4c0caa00 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c0cf870 con 0x7ffa4c06c0b0 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.326+0000 7ffa50bc7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4c159f70 con 0x7ffa4c1544b0 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.327+0000 7ffa49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.327+0000 7ffa49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50308/0 (socket says 172.21.15.129:50308) 2024-03-23T15:50:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.327+0000 7ffa49d9b640 1 -- 172.21.15.129:0/2853926122 learned_addr learned my addr 172.21.15.129:0/2853926122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.333+0000 7ffa49d9b640 1 -- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa4c1544b0 msgr2=0x7ffa4c154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:39.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.333+0000 7ffa49d9b640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa4c1544b0 0x7ffa4c154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.333+0000 7ffa49d9b640 1 -- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa4c06c0b0 msgr2=0x7ffa4c1484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:39.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.333+0000 7ffa49d9b640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa4c06c0b0 0x7ffa4c1484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.333+0000 7ffa49d9b640 1 -- 172.21.15.129:0/2853926122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa400094d0 con 0x7ffa4c0caa00 2024-03-23T15:50:39.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.335+0000 7ffa49d9b640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c156490 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffa40000c00 tx=0x7ffa40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:39.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.338+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4002f820 con 0x7ffa4c0caa00 2024-03-23T15:50:39.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.338+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4c0cf590 con 0x7ffa4c0caa00 2024-03-23T15:50:39.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.338+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4c157420 con 0x7ffa4c0caa00 2024-03-23T15:50:39.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.339+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa4c075740 con 0x7ffa4c0caa00 2024-03-23T15:50:39.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.340+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4004b460 con 0x7ffa4c0caa00 2024-03-23T15:50:39.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.340+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4004b7d0 con 0x7ffa4c0caa00 2024-03-23T15:50:39.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.621+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffa4004b930 con 0x7ffa4c0caa00 2024-03-23T15:50:39.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.622+0000 7ffa3affd640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:39.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.622+0000 7ffa4a59c640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:39.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.623+0000 7ffa4a59c640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009670 tx=0x7ffa3c006bf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:39.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.636+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ffa4003c070 con 0x7ffa4c0caa00 2024-03-23T15:50:39.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.636+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa400787d0 con 0x7ffa4c0caa00 2024-03-23T15:50:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.876+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa4c0cba10 con 0x7ffa4c0caa00 2024-03-23T15:50:39.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.891+0000 7ffa3affd640 1 -- 172.21.15.129:0/2853926122 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ffa4007be90 con 0x7ffa4c0caa00 2024-03-23T15:50:39.895 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:39.895 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:39.895 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2c06b810 msgr2=0x7ffa2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009670 tx=0x7ffa3c006bf0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 msgr2=0x7ffa4c156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c156490 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffa40000c00 tx=0x7ffa40004730 comp rx=0 tx=0).stop 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 shutdown_connections 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa4c1544b0 0x7ffa4c154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4c0caa00 0x7ffa4c156490 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 --2- 172.21.15.129:0/2853926122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa4c06c0b0 0x7ffa4c1484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.894+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 >> 172.21.15.129:0/2853926122 conn(0x7ffa4c061b50 msgr2=0x7ffa4c0b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.895+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 shutdown_connections 2024-03-23T15:50:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:39.895+0000 7ffa50bc7640 1 -- 172.21.15.129:0/2853926122 wait complete. 2024-03-23T15:50:40.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:40.513 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:41.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:41.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.614+0000 7efcc9393640 1 Processor -- start 2024-03-23T15:50:41.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.614+0000 7efcc9393640 1 -- start start 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.615+0000 7efcc9393640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efcc41544b0 0x7efcc415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.615+0000 7efcc9393640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efcc40caa00 0x7efcc406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.615+0000 7efcc9393640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc406c0b0 0x7efcc40c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.615+0000 7efcc9393640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7efcc40c5be0 con 0x7efcc41544b0 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.615+0000 7efcc9393640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efcc40cbcb0 con 0x7efcc40caa00 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.615+0000 7efcc9393640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efcc415b280 con 0x7efcc406c0b0 2024-03-23T15:50:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.616+0000 7efcc2ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efcc41544b0 0x7efcc415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.616+0000 7efcc27fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efcc40caa00 0x7efcc406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.616+0000 7efcc37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc406c0b0 0x7efcc40c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.616+0000 7efcc2ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efcc41544b0 0x7efcc415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44266/0 (socket says 172.21.15.129:44266) 2024-03-23T15:50:41.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.616+0000 7efcc2ffd640 1 -- 172.21.15.129:0/376800940 learned_addr learned my addr 172.21.15.129:0/376800940 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:41.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.619+0000 7efcc2ffd640 1 -- 172.21.15.129:0/376800940 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc406c0b0 msgr2=0x7efcc40c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:41.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.619+0000 7efcc2ffd640 1 --2- 172.21.15.129:0/376800940 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc406c0b0 0x7efcc40c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.619+0000 7efcc2ffd640 1 -- 172.21.15.129:0/376800940 >> v2:172.21.15.157:3300/0 conn(0x7efcc40caa00 msgr2=0x7efcc406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:41.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.619+0000 7efcc2ffd640 1 --2- 172.21.15.129:0/376800940 >> v2:172.21.15.157:3300/0 conn(0x7efcc40caa00 0x7efcc406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.619+0000 7efcc2ffd640 1 -- 172.21.15.129:0/376800940 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc415a9e0 con 0x7efcc41544b0 2024-03-23T15:50:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.621+0000 7efcc2ffd640 1 --2- 172.21.15.129:0/376800940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc41544b0 0x7efcc415a2e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efcc406ce60 tx=0x7efcb402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bb641c6e65eb04 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:41.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.623+0000 7efcc1ffb640 1 -- 172.21.15.129:0/376800940 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4030860 con 0x7efcc41544b0 2024-03-23T15:50:41.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.625+0000 7efcc1ffb640 1 -- 172.21.15.129:0/376800940 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb40309c0 con 0x7efcc41544b0 2024-03-23T15:50:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc1ffb640 1 -- 172.21.15.129:0/376800940 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4030d50 con 0x7efcc41544b0 2024-03-23T15:50:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 -- 172.21.15.129:0/376800940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc41544b0 msgr2=0x7efcc415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 --2- 172.21.15.129:0/376800940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc41544b0 0x7efcc415a2e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efcc406ce60 tx=0x7efcb402b9a0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 -- 172.21.15.129:0/376800940 shutdown_connections 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 --2- 172.21.15.129:0/376800940 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc406c0b0 0x7efcc40c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 --2- 172.21.15.129:0/376800940 >> v2:172.21.15.157:3300/0 conn(0x7efcc40caa00 0x7efcc406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 --2- 172.21.15.129:0/376800940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc41544b0 0x7efcc415a2e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 -- 172.21.15.129:0/376800940 >> 172.21.15.129:0/376800940 conn(0x7efcc4061b50 msgr2=0x7efcc4061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 -- 172.21.15.129:0/376800940 shutdown_connections 2024-03-23T15:50:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.626+0000 7efcc9393640 1 -- 172.21.15.129:0/376800940 wait complete. 2024-03-23T15:50:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.628+0000 7efcc9393640 1 Processor -- start 2024-03-23T15:50:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.628+0000 7efcc9393640 1 -- start start 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc9393640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 0x7efcc4156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc9393640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc40caa00 0x7efcc4159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc9393640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc41544b0 0x7efcc41f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc9393640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efcc4157030 con 0x7efcc406c0b0 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc9393640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efcc41f4220 con 0x7efcc41544b0 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc9393640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efcc41f1390 con 0x7efcc40caa00 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 0x7efcc4156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc27fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc40caa00 0x7efcc4159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.630+0000 7efcc37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc41544b0 0x7efcc41f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.631+0000 7efcc2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 0x7efcc4156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44268/0 (socket says 172.21.15.129:44268) 2024-03-23T15:50:41.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.631+0000 7efcc2ffd640 1 -- 172.21.15.129:0/4067473132 learned_addr learned my addr 172.21.15.129:0/4067473132 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.634+0000 7efcc2ffd640 1 -- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc40caa00 msgr2=0x7efcc4159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.634+0000 7efcc2ffd640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc40caa00 0x7efcc4159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.634+0000 7efcc2ffd640 1 -- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc41544b0 msgr2=0x7efcc41f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.634+0000 7efcc2ffd640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc41544b0 0x7efcc41f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.634+0000 7efcc2ffd640 1 -- 172.21.15.129:0/4067473132 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb40094d0 con 0x7efcc406c0b0 2024-03-23T15:50:41.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.636+0000 7efcc2ffd640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 0x7efcc4156af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcb402b9e0 tx=0x7efcb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.640+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4039a90 con 0x7efcc406c0b0 2024-03-23T15:50:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.640+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb4039c10 con 0x7efcc406c0b0 2024-03-23T15:50:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.640+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4043410 con 0x7efcc406c0b0 2024-03-23T15:50:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.640+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc41f1670 con 0x7efcc406c0b0 2024-03-23T15:50:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.640+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc41f1c30 con 0x7efcc406c0b0 2024-03-23T15:50:41.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.641+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc4075740 con 0x7efcc406c0b0 2024-03-23T15:50:41.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.669+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efcb404c3b0 con 0x7efcc406c0b0 2024-03-23T15:50:41.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.670+0000 7efcab7fe640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efcb006b860 0x7efcb006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:41.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.670+0000 7efcc27fc640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efcb006b860 0x7efcb006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:41.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.671+0000 7efcc27fc640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efcb006b860 0x7efcb006dd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efcc4159d50 tx=0x7efcb8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:41.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.674+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efcb407b720 con 0x7efcc406c0b0 2024-03-23T15:50:41.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:41.919+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcb40779c0 con 0x7efcc406c0b0 2024-03-23T15:50:42.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.171+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcc4157d00 con 0x7efcc406c0b0 2024-03-23T15:50:42.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.186+0000 7efcab7fe640 1 -- 172.21.15.129:0/4067473132 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7efcb4034090 con 0x7efcc406c0b0 2024-03-23T15:50:42.189 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:42.189 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efcb006b860 msgr2=0x7efcb006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efcb006b860 0x7efcb006dd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efcc4159d50 tx=0x7efcb8009190 comp rx=0 tx=0).stop 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 msgr2=0x7efcc4156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 0x7efcc4156af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcb402b9e0 tx=0x7efcb4004730 comp rx=0 tx=0).stop 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 shutdown_connections 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efcb006b860 0x7efcb006dd20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efcc41544b0 0x7efcc41f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efcc40caa00 0x7efcc4159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 --2- 172.21.15.129:0/4067473132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcc406c0b0 0x7efcc4156af0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 >> 172.21.15.129:0/4067473132 conn(0x7efcc4061b50 msgr2=0x7efcc40b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 shutdown_connections 2024-03-23T15:50:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:42.189+0000 7efcc9393640 1 -- 172.21.15.129:0/4067473132 wait complete. 2024-03-23T15:50:42.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:42.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:43.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:43.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.907+0000 7ff7354e8640 1 Processor -- start 2024-03-23T15:50:43.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.907+0000 7ff7354e8640 1 -- start start 2024-03-23T15:50:43.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.908+0000 7ff7354e8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff7301544b0 0x7ff73015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:43.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.908+0000 7ff7354e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 0x7ff7300c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:43.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.908+0000 7ff7354e8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7300cbcb0 con 0x7ff7300caa00 2024-03-23T15:50:43.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.908+0000 7ff7354e8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff7300c5be0 con 0x7ff7301544b0 2024-03-23T15:50:43.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.908+0000 7ff7354e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7300c69b0 con 0x7ff7300c6000 2024-03-23T15:50:43.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.909+0000 7ff72f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 0x7ff7300c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:43.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.909+0000 7ff72effd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff7301544b0 0x7ff73015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:43.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.909+0000 7ff72f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 0x7ff7300c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50476/0 (socket says 172.21.15.129:50476) 2024-03-23T15:50:43.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.909+0000 7ff72f7fe640 1 -- 172.21.15.129:0/4219303357 learned_addr learned my addr 172.21.15.129:0/4219303357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:43.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.912+0000 7ff72f7fe640 1 -- 172.21.15.129:0/4219303357 >> v2:172.21.15.157:3300/0 conn(0x7ff7301544b0 msgr2=0x7ff73015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:43.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.912+0000 7ff72f7fe640 1 --2- 172.21.15.129:0/4219303357 >> v2:172.21.15.157:3300/0 conn(0x7ff7301544b0 0x7ff73015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:43.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.912+0000 7ff72f7fe640 1 -- 172.21.15.129:0/4219303357 >> v1:172.21.15.157:6789/0 conn(0x7ff7300caa00 legacy=0x7ff73006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:43.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.912+0000 7ff72f7fe640 1 -- 172.21.15.129:0/4219303357 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73015b280 con 0x7ff7300c6000 2024-03-23T15:50:43.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.914+0000 7ff72f7fe640 1 --2- 172.21.15.129:0/4219303357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 0x7ff7300c6470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff728009e50 tx=0x7ff72802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95976f08bb9a1897 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:43.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.918+0000 7ff72e7fc640 1 -- 172.21.15.129:0/4219303357 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72802cb00 con 0x7ff7300c6000 2024-03-23T15:50:43.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.918+0000 7ff72e7fc640 1 -- 172.21.15.129:0/4219303357 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff72802cc60 con 0x7ff7300c6000 2024-03-23T15:50:43.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.918+0000 7ff72e7fc640 1 -- 172.21.15.129:0/4219303357 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7280343f0 con 0x7ff7300c6000 2024-03-23T15:50:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 -- 172.21.15.129:0/4219303357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 msgr2=0x7ff7300c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 --2- 172.21.15.129:0/4219303357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 0x7ff7300c6470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff728009e50 tx=0x7ff72802bf60 comp rx=0 tx=0).stop 2024-03-23T15:50:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 -- 172.21.15.129:0/4219303357 shutdown_connections 2024-03-23T15:50:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 --2- 172.21.15.129:0/4219303357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300c6000 0x7ff7300c6470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 --2- 172.21.15.129:0/4219303357 >> v2:172.21.15.157:3300/0 conn(0x7ff7301544b0 0x7ff73015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 -- 172.21.15.129:0/4219303357 >> 172.21.15.129:0/4219303357 conn(0x7ff730061b50 msgr2=0x7ff730061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 -- 172.21.15.129:0/4219303357 shutdown_connections 2024-03-23T15:50:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.919+0000 7ff7354e8640 1 -- 172.21.15.129:0/4219303357 wait complete. 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.920+0000 7ff7354e8640 1 Processor -- start 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.920+0000 7ff7354e8640 1 -- start start 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff7354e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7300c6000 0x7ff7300c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff7354e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300caa00 0x7ff730141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff7354e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 0x7ff73006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff7354e8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff730159860 con 0x7ff7300c6000 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff7354e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7301efff0 con 0x7ff7301544b0 2024-03-23T15:50:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff7354e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7300c9260 con 0x7ff7300caa00 2024-03-23T15:50:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff72f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 0x7ff73006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff727fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300caa00 0x7ff730141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.921+0000 7ff72effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7300c6000 0x7ff7300c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.922+0000 7ff72f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 0x7ff73006bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:50022/0 (socket says 172.21.15.129:50022) 2024-03-23T15:50:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.922+0000 7ff72f7fe640 1 -- 172.21.15.129:0/3086592075 learned_addr learned my addr 172.21.15.129:0/3086592075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:43.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.925+0000 7ff72f7fe640 1 -- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300caa00 msgr2=0x7ff730141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:43.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.925+0000 7ff72f7fe640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300caa00 0x7ff730141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:43.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.925+0000 7ff72f7fe640 1 -- 172.21.15.129:0/3086592075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7300c6000 msgr2=0x7ff7300c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:43.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.925+0000 7ff72f7fe640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7300c6000 0x7ff7300c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:43.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.925+0000 7ff72f7fe640 1 -- 172.21.15.129:0/3086592075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff728009b30 con 0x7ff7301544b0 2024-03-23T15:50:43.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.928+0000 7ff72f7fe640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 0x7ff73006bb50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff728036040 tx=0x7ff728004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:43.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.932+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728009510 con 0x7ff7301544b0 2024-03-23T15:50:43.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.932+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff72803e450 con 0x7ff7301544b0 2024-03-23T15:50:43.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.932+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7300c8780 con 0x7ff7301544b0 2024-03-23T15:50:43.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.932+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72803e7e0 con 0x7ff7301544b0 2024-03-23T15:50:43.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.932+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7300c9e40 con 0x7ff7301544b0 2024-03-23T15:50:43.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.933+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7300ca750 con 0x7ff7301544b0 2024-03-23T15:50:43.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.952+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff72803e940 con 0x7ff7301544b0 2024-03-23T15:50:43.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.953+0000 7ff7277fe640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff71006b860 0x7ff71006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:43.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.953+0000 7ff72effd640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff71006b860 0x7ff71006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:43.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:43.953+0000 7ff72effd640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff71006b860 0x7ff71006dd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff720009aa0 tx=0x7ff720009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:44.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.195+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff72807be70 con 0x7ff7301544b0 2024-03-23T15:50:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.205+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff728078110 con 0x7ff7301544b0 2024-03-23T15:50:44.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.422+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff730075740 con 0x7ff7301544b0 2024-03-23T15:50:44.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.443+0000 7ff7277fe640 1 -- 172.21.15.129:0/3086592075 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff72807b7d0 con 0x7ff7301544b0 2024-03-23T15:50:44.447 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:44.447 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:44.447 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff71006b860 msgr2=0x7ff71006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff71006b860 0x7ff71006dd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff720009aa0 tx=0x7ff720009190 comp rx=0 tx=0).stop 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 msgr2=0x7ff73006bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 0x7ff73006bb50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff728036040 tx=0x7ff728004a40 comp rx=0 tx=0).stop 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 shutdown_connections 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff71006b860 0x7ff71006dd20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7301544b0 0x7ff73006bb50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7300caa00 0x7ff730141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 --2- 172.21.15.129:0/3086592075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7300c6000 0x7ff7300c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 >> 172.21.15.129:0/3086592075 conn(0x7ff730061b50 msgr2=0x7ff73014a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 shutdown_connections 2024-03-23T15:50:44.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:44.445+0000 7ff7354e8640 1 -- 172.21.15.129:0/3086592075 wait complete. 2024-03-23T15:50:45.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:45.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:46.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.174+0000 7f765a54f640 1 Processor -- start 2024-03-23T15:50:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.174+0000 7f765a54f640 1 -- start start 2024-03-23T15:50:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.175+0000 7f765a54f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f765415a2e0 0x7f765406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.175+0000 7f765a54f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f765406c0b0 0x7f76540c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.175+0000 7f765a54f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76540cbcb0 con 0x7f76541544b0 2024-03-23T15:50:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.175+0000 7f765a54f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f76540c5be0 con 0x7f765406c0b0 2024-03-23T15:50:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.175+0000 7f765a54f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f765415b280 con 0x7f765415a2e0 2024-03-23T15:50:46.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.176+0000 7f7658aec640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f765406c0b0 0x7f76540c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:46.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.176+0000 7f76537fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f765415a2e0 0x7f765406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:46.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.176+0000 7f7658aec640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f765406c0b0 0x7f76540c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44284/0 (socket says 172.21.15.129:44284) 2024-03-23T15:50:46.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.176+0000 7f7658aec640 1 -- 172.21.15.129:0/883637961 learned_addr learned my addr 172.21.15.129:0/883637961 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.178+0000 7f7658aec640 1 -- 172.21.15.129:0/883637961 >> v2:172.21.15.157:3300/0 conn(0x7f765415a2e0 msgr2=0x7f765406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.178+0000 7f7658aec640 1 --2- 172.21.15.129:0/883637961 >> v2:172.21.15.157:3300/0 conn(0x7f765415a2e0 0x7f765406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.178+0000 7f7658aec640 1 -- 172.21.15.129:0/883637961 >> v1:172.21.15.157:6789/0 conn(0x7f76541544b0 legacy=0x7f76540caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.178+0000 7f7658aec640 1 -- 172.21.15.129:0/883637961 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765415a9e0 con 0x7f765406c0b0 2024-03-23T15:50:46.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.180+0000 7f7658aec640 1 --2- 172.21.15.129:0/883637961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540c6000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f765406ce60 tx=0x7f764c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a796f84eb32639 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f7652ffd640 1 -- 172.21.15.129:0/883637961 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c02cb20 con 0x7f765406c0b0 2024-03-23T15:50:46.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f7652ffd640 1 -- 172.21.15.129:0/883637961 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f764c02cc80 con 0x7f765406c0b0 2024-03-23T15:50:46.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f7652ffd640 1 -- 172.21.15.129:0/883637961 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c0343f0 con 0x7f765406c0b0 2024-03-23T15:50:46.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 -- 172.21.15.129:0/883637961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 msgr2=0x7f76540c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 --2- 172.21.15.129:0/883637961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540c6000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f765406ce60 tx=0x7f764c02bf80 comp rx=0 tx=0).stop 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 -- 172.21.15.129:0/883637961 shutdown_connections 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 --2- 172.21.15.129:0/883637961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540c6000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 --2- 172.21.15.129:0/883637961 >> v2:172.21.15.157:3300/0 conn(0x7f765415a2e0 0x7f765406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 -- 172.21.15.129:0/883637961 >> 172.21.15.129:0/883637961 conn(0x7f7654061b50 msgr2=0x7f7654061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 -- 172.21.15.129:0/883637961 shutdown_connections 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.185+0000 7f765a54f640 1 -- 172.21.15.129:0/883637961 wait complete. 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.187+0000 7f765a54f640 1 Processor -- start 2024-03-23T15:50:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.187+0000 7f765a54f640 1 -- start start 2024-03-23T15:50:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.187+0000 7f765a54f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.187+0000 7f765a54f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76541544b0 0x7f76540ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f765a54f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f765415a2e0 0x7f76540ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f765a54f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76540d0660 con 0x7f765406c0b0 2024-03-23T15:50:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f765a54f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7654157b20 con 0x7f76541544b0 2024-03-23T15:50:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f765a54f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f76541575a0 con 0x7f765415a2e0 2024-03-23T15:50:46.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f7653fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:46.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f76537fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76541544b0 0x7f76540ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:46.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f7658aec640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f765415a2e0 0x7f76540ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:46.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f7653fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44300/0 (socket says 172.21.15.129:44300) 2024-03-23T15:50:46.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.188+0000 7f7653fff640 1 -- 172.21.15.129:0/3584394995 learned_addr learned my addr 172.21.15.129:0/3584394995 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.192+0000 7f7653fff640 1 -- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f765415a2e0 msgr2=0x7f76540ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.192+0000 7f7653fff640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f765415a2e0 0x7f76540ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.192+0000 7f7653fff640 1 -- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76541544b0 msgr2=0x7f76540ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.192+0000 7f7653fff640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76541544b0 0x7f76540ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.192+0000 7f7653fff640 1 -- 172.21.15.129:0/3584394995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764c009b30 con 0x7f765406c0b0 2024-03-23T15:50:46.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.194+0000 7f7653fff640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540cf950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f764400c410 tx=0x7f764400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:46.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.198+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440035f0 con 0x7f765406c0b0 2024-03-23T15:50:46.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.198+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7644003750 con 0x7f765406c0b0 2024-03-23T15:50:46.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.198+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76540d0330 con 0x7f765406c0b0 2024-03-23T15:50:46.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.198+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644003b40 con 0x7f765406c0b0 2024-03-23T15:50:46.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.198+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7654155a00 con 0x7f765406c0b0 2024-03-23T15:50:46.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.199+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7654068d90 con 0x7f765406c0b0 2024-03-23T15:50:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.464+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7644003ca0 con 0x7f765406c0b0 2024-03-23T15:50:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.464+0000 7f7650ff9640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f764006b860 0x7f764006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.464+0000 7f76537fe640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f764006b860 0x7f764006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:46.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.465+0000 7f76537fe640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f764006b860 0x7f764006dd20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f76540cbb40 tx=0x7f7648009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:46.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.478+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f76440010c0 con 0x7f765406c0b0 2024-03-23T15:50:46.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.478+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7644054740 con 0x7f765406c0b0 2024-03-23T15:50:46.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.711+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7654153090 con 0x7f765406c0b0 2024-03-23T15:50:46.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.723+0000 7f7650ff9640 1 -- 172.21.15.129:0/3584394995 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f7644017070 con 0x7f765406c0b0 2024-03-23T15:50:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:46.726 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:46.727 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f764006b860 msgr2=0x7f764006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f764006b860 0x7f764006dd20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f76540cbb40 tx=0x7f7648009190 comp rx=0 tx=0).stop 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 msgr2=0x7f76540cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540cf950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f764400c410 tx=0x7f764400c8d0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 shutdown_connections 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f765415a2e0 0x7f76540ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f764006b860 0x7f764006dd20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76541544b0 0x7f76540ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 --2- 172.21.15.129:0/3584394995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765406c0b0 0x7f76540cf950 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 >> 172.21.15.129:0/3584394995 conn(0x7f7654061b50 msgr2=0x7f76541534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:46.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 shutdown_connections 2024-03-23T15:50:46.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:46.726+0000 7f765a54f640 1 -- 172.21.15.129:0/3584394995 wait complete. 2024-03-23T15:50:47.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:47.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:48.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:48.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.446+0000 7ffb11c15640 1 Processor -- start 2024-03-23T15:50:48.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.446+0000 7ffb11c15640 1 -- start start 2024-03-23T15:50:48.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.447+0000 7ffb11c15640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb0c15dba0 0x7ffb0c06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:48.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.447+0000 7ffb11c15640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffb0c15df80 0x7ffb0c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:48.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.447+0000 7ffb11c15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb0c070640 con 0x7ffb0c0cca30 2024-03-23T15:50:48.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.447+0000 7ffb11c15640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0c0ccea0 con 0x7ffb0c15dba0 2024-03-23T15:50:48.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.447+0000 7ffb11c15640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0c0c4c00 con 0x7ffb0c15df80 2024-03-23T15:50:48.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.448+0000 7ffb0b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb0c15dba0 0x7ffb0c06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:48.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.449+0000 7ffb0affd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ffb0c0cca30 0x7ffb0c06ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:34660/0 (socket says 172.21.15.129:34660) 2024-03-23T15:50:48.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.449+0000 7ffb0affd640 1 -- 172.21.15.129:0/2043538117 learned_addr learned my addr 172.21.15.129:0/2043538117 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:50:48.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.451+0000 7ffb0bfff640 1 --2- 172.21.15.129:0/2043538117 >> v2:172.21.15.157:3300/0 conn(0x7ffb0c15df80 0x7ffb0c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.454+0000 7ffb0b7fe640 1 -- 172.21.15.129:0/2043538117 >> v2:172.21.15.157:3300/0 conn(0x7ffb0c15df80 msgr2=0x7ffb0c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.454+0000 7ffb0b7fe640 1 --2- 172.21.15.129:0/2043538117 >> v2:172.21.15.157:3300/0 conn(0x7ffb0c15df80 0x7ffb0c1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.454+0000 7ffb0b7fe640 1 -- 172.21.15.129:0/2043538117 >> v1:172.21.15.129:6789/0 conn(0x7ffb0c0cca30 legacy=0x7ffb0c06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.454+0000 7ffb0b7fe640 1 -- 172.21.15.129:0/2043538117 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0c06c240 con 0x7ffb0c15dba0 2024-03-23T15:50:48.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.458+0000 7ffb0b7fe640 1 --2- 172.21.15.129:0/2043538117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c06cf70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0097a0 tx=0x7ffafc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3a99530f667caf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:48.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb0a7fc640 1 -- 172.21.15.129:0/2043538117 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc030610 con 0x7ffb0c15dba0 2024-03-23T15:50:48.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb0a7fc640 1 -- 172.21.15.129:0/2043538117 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffafc030770 con 0x7ffb0c15dba0 2024-03-23T15:50:48.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb0a7fc640 1 -- 172.21.15.129:0/2043538117 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc030b00 con 0x7ffb0c15dba0 2024-03-23T15:50:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb11c15640 1 -- 172.21.15.129:0/2043538117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 msgr2=0x7ffb0c06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2043538117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c06cf70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0097a0 tx=0x7ffafc02b960 comp rx=0 tx=0).stop 2024-03-23T15:50:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb11c15640 1 -- 172.21.15.129:0/2043538117 shutdown_connections 2024-03-23T15:50:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2043538117 >> v2:172.21.15.157:3300/0 conn(0x7ffb0c15df80 0x7ffb0c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2043538117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c06cf70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.479+0000 7ffb11c15640 1 -- 172.21.15.129:0/2043538117 >> 172.21.15.129:0/2043538117 conn(0x7ffb0c071300 msgr2=0x7ffb0c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:48.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.480+0000 7ffb11c15640 1 -- 172.21.15.129:0/2043538117 shutdown_connections 2024-03-23T15:50:48.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.480+0000 7ffb11c15640 1 -- 172.21.15.129:0/2043538117 wait complete. 2024-03-23T15:50:48.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.481+0000 7ffb11c15640 1 Processor -- start 2024-03-23T15:50:48.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.482+0000 7ffb11c15640 1 -- start start 2024-03-23T15:50:48.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb11c15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb0c0cca30 0x7ffb0c1597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb11c15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c1474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb11c15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb0c15df80 0x7ffb0c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb11c15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c159d20 con 0x7ffb0c15dba0 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb11c15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c15a900 con 0x7ffb0c15df80 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb11c15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffb0c15aea0 con 0x7ffb0c0cca30 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb0affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c1474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb0b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb0c0cca30 0x7ffb0c1597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.483+0000 7ffb0bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb0c15df80 0x7ffb0c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.484+0000 7ffb0affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c1474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44314/0 (socket says 172.21.15.129:44314) 2024-03-23T15:50:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.484+0000 7ffb0affd640 1 -- 172.21.15.129:0/2199837697 learned_addr learned my addr 172.21.15.129:0/2199837697 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.487+0000 7ffb0affd640 1 -- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb0c0cca30 msgr2=0x7ffb0c1597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.487+0000 7ffb0affd640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb0c0cca30 0x7ffb0c1597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.487+0000 7ffb0affd640 1 -- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb0c15df80 msgr2=0x7ffb0c158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.487+0000 7ffb0affd640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb0c15df80 0x7ffb0c158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.487+0000 7ffb0affd640 1 -- 172.21.15.129:0/2199837697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc009400 con 0x7ffb0c15dba0 2024-03-23T15:50:48.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.490+0000 7ffb0affd640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c1474f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c400 tx=0x7ffb0000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.494+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb000074e0 con 0x7ffb0c15dba0 2024-03-23T15:50:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.494+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb00007640 con 0x7ffb0c15dba0 2024-03-23T15:50:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.494+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb000059a0 con 0x7ffb0c15dba0 2024-03-23T15:50:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.494+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0c1f4440 con 0x7ffb0c15dba0 2024-03-23T15:50:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.494+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0c158190 con 0x7ffb0c15dba0 2024-03-23T15:50:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.495+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb0c1586b0 con 0x7ffb0c15dba0 2024-03-23T15:50:48.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.757+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffb00005b00 con 0x7ffb0c15dba0 2024-03-23T15:50:48.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.757+0000 7ffaf3fff640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffaf806b810 0x7ffaf806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:48.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.758+0000 7ffb0b7fe640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffaf806b810 0x7ffaf806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.758+0000 7ffb0b7fe640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffaf806b810 0x7ffaf806dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffafc036000 tx=0x7ffafc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.779+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ffb00058560 con 0x7ffb0c15dba0 2024-03-23T15:50:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:48.779+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffb00054800 con 0x7ffb0c15dba0 2024-03-23T15:50:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.008+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb0c1586b0 con 0x7ffb0c15dba0 2024-03-23T15:50:49.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.044+0000 7ffaf3fff640 1 -- 172.21.15.129:0/2199837697 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ffb0c1586b0 con 0x7ffb0c15dba0 2024-03-23T15:50:49.047 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:49.047 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:49.048 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:49.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffaf806b810 msgr2=0x7ffaf806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffaf806b810 0x7ffaf806dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffafc036000 tx=0x7ffafc008030 comp rx=0 tx=0).stop 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 msgr2=0x7ffb0c1474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c1474f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000c400 tx=0x7ffb0000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 shutdown_connections 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffaf806b810 0x7ffaf806dcd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb0c15df80 0x7ffb0c158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb0c15dba0 0x7ffb0c1474f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 --2- 172.21.15.129:0/2199837697 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb0c0cca30 0x7ffb0c1597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.046+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 >> 172.21.15.129:0/2199837697 conn(0x7ffb0c071300 msgr2=0x7ffb0c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.047+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 shutdown_connections 2024-03-23T15:50:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:49.047+0000 7ffb11c15640 1 -- 172.21.15.129:0/2199837697 wait complete. 2024-03-23T15:50:49.666 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:49.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:50.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:50.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.766+0000 7fd2e07f7640 1 Processor -- start 2024-03-23T15:50:50.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.766+0000 7fd2e07f7640 1 -- start start 2024-03-23T15:50:50.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.767+0000 7fd2e07f7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2d81544b0 0x7fd2d815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:50.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.767+0000 7fd2e07f7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:50.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.767+0000 7fd2e07f7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d80cbcb0 con 0x7fd2d80caa00 2024-03-23T15:50:50.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.767+0000 7fd2e07f7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d80c5be0 con 0x7fd2d81544b0 2024-03-23T15:50:50.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.767+0000 7fd2e07f7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80c69b0 con 0x7fd2d80c6000 2024-03-23T15:50:50.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.768+0000 7fd2ded94640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:50.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.768+0000 7fd2de593640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2d81544b0 0x7fd2d815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:50.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.768+0000 7fd2ded94640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:56794/0 (socket says 172.21.15.129:56794) 2024-03-23T15:50:50.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.768+0000 7fd2ded94640 1 -- 172.21.15.129:0/3342692485 learned_addr learned my addr 172.21.15.129:0/3342692485 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:50.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.771+0000 7fd2ded94640 1 -- 172.21.15.129:0/3342692485 >> v2:172.21.15.157:3300/0 conn(0x7fd2d81544b0 msgr2=0x7fd2d815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:50.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.771+0000 7fd2ded94640 1 --2- 172.21.15.129:0/3342692485 >> v2:172.21.15.157:3300/0 conn(0x7fd2d81544b0 0x7fd2d815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:50.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.771+0000 7fd2ded94640 1 -- 172.21.15.129:0/3342692485 >> v1:172.21.15.157:6789/0 conn(0x7fd2d80caa00 legacy=0x7fd2d806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:50.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.771+0000 7fd2ded94640 1 -- 172.21.15.129:0/3342692485 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d815b280 con 0x7fd2d80c6000 2024-03-23T15:50:50.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.773+0000 7fd2ded94640 1 --2- 172.21.15.129:0/3342692485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c6470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009e50 tx=0x7fd2d002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ccc5423425de11 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:50.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2dd591640 1 -- 172.21.15.129:0/3342692485 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d002cb00 con 0x7fd2d80c6000 2024-03-23T15:50:50.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2dd591640 1 -- 172.21.15.129:0/3342692485 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d002cc60 con 0x7fd2d80c6000 2024-03-23T15:50:50.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2dd591640 1 -- 172.21.15.129:0/3342692485 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00343f0 con 0x7fd2d80c6000 2024-03-23T15:50:50.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 -- 172.21.15.129:0/3342692485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 msgr2=0x7fd2d80c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/3342692485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c6470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009e50 tx=0x7fd2d002bf60 comp rx=0 tx=0).stop 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 -- 172.21.15.129:0/3342692485 shutdown_connections 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/3342692485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c6470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/3342692485 >> v2:172.21.15.157:3300/0 conn(0x7fd2d81544b0 0x7fd2d815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 -- 172.21.15.129:0/3342692485 >> 172.21.15.129:0/3342692485 conn(0x7fd2d8061b50 msgr2=0x7fd2d8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 -- 172.21.15.129:0/3342692485 shutdown_connections 2024-03-23T15:50:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.778+0000 7fd2e07f7640 1 -- 172.21.15.129:0/3342692485 wait complete. 2024-03-23T15:50:50.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.779+0000 7fd2e07f7640 1 Processor -- start 2024-03-23T15:50:50.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.779+0000 7fd2e07f7640 1 -- start start 2024-03-23T15:50:50.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.780+0000 7fd2e07f7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:50.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.780+0000 7fd2e07f7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2d80caa00 0x7fd2d8141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:50.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.780+0000 7fd2e07f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d81544b0 0x7fd2d806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.780+0000 7fd2e07f7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d8159860 con 0x7fd2d81544b0 2024-03-23T15:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.780+0000 7fd2e07f7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81efff0 con 0x7fd2d80caa00 2024-03-23T15:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.780+0000 7fd2e07f7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80c9260 con 0x7fd2d80c6000 2024-03-23T15:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.781+0000 7fd2de593640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.781+0000 7fd2ddd92640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2d80caa00 0x7fd2d8141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.781+0000 7fd2de593640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:56806/0 (socket says 172.21.15.129:56806) 2024-03-23T15:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.781+0000 7fd2de593640 1 -- 172.21.15.129:0/1847964030 learned_addr learned my addr 172.21.15.129:0/1847964030 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.781+0000 7fd2ded94640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d81544b0 0x7fd2d806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.786+0000 7fd2de593640 1 -- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2d80caa00 msgr2=0x7fd2d8141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.786+0000 7fd2de593640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2d80caa00 0x7fd2d8141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.786+0000 7fd2de593640 1 -- 172.21.15.129:0/1847964030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d81544b0 msgr2=0x7fd2d806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.786+0000 7fd2de593640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d81544b0 0x7fd2d806bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.786+0000 7fd2de593640 1 -- 172.21.15.129:0/1847964030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d0009b30 con 0x7fd2d80c6000 2024-03-23T15:50:50.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.790+0000 7fd2de593640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c8d20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400e870 tx=0x7fd2d400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:50.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.798+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d40094f0 con 0x7fd2d80c6000 2024-03-23T15:50:50.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.798+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d400c940 con 0x7fd2d80c6000 2024-03-23T15:50:50.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.798+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d400ccf0 con 0x7fd2d80c6000 2024-03-23T15:50:50.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.798+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d80c8780 con 0x7fd2d80c6000 2024-03-23T15:50:50.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.798+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d80c9e40 con 0x7fd2d80c6000 2024-03-23T15:50:50.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:50.799+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d80ca770 con 0x7fd2d80c6000 2024-03-23T15:50:51.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.066+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2d4013470 con 0x7fd2d80c6000 2024-03-23T15:50:51.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.067+0000 7fd2ceffd640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2bc06b860 0x7fd2bc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:51.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.067+0000 7fd2ddd92640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2bc06b860 0x7fd2bc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:51.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.068+0000 7fd2ddd92640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2bc06b860 0x7fd2bc06dd20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8149240 tx=0x7fd2c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:51.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.075+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd2d4014e40 con 0x7fd2d80c6000 2024-03-23T15:50:51.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.091+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2d4055470 con 0x7fd2d80c6000 2024-03-23T15:50:51.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.316+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2d8075740 con 0x7fd2d80c6000 2024-03-23T15:50:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.337+0000 7fd2ceffd640 1 -- 172.21.15.129:0/1847964030 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd2d4021090 con 0x7fd2d80c6000 2024-03-23T15:50:51.341 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:51.341 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2bc06b860 msgr2=0x7fd2bc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2bc06b860 0x7fd2bc06dd20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8149240 tx=0x7fd2c8009040 comp rx=0 tx=0).stop 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 msgr2=0x7fd2d80c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c8d20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400e870 tx=0x7fd2d400ed30 comp rx=0 tx=0).stop 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 shutdown_connections 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d81544b0 0x7fd2d806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2bc06b860 0x7fd2bc06dd20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2d80caa00 0x7fd2d8141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:51.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 --2- 172.21.15.129:0/1847964030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2d80c6000 0x7fd2d80c8d20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 >> 172.21.15.129:0/1847964030 conn(0x7fd2d8061b50 msgr2=0x7fd2d814a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 shutdown_connections 2024-03-23T15:50:51.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:51.340+0000 7fd2e07f7640 1 -- 172.21.15.129:0/1847964030 wait complete. 2024-03-23T15:50:51.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:51.960 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:52.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:53.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 Processor -- start 2024-03-23T15:50:53.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 -- start start 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f96cc15a2e0 0x7f96cc06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f96cc06c0b0 0x7f96cc0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96cc0cbcb0 con 0x7f96cc1544b0 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f96cc0c5be0 con 0x7f96cc15a2e0 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.061+0000 7f96d1aeb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f96cc15b280 con 0x7f96cc06c0b0 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.062+0000 7f96caffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f96cc15a2e0 0x7f96cc06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:53.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.062+0000 7f96cbfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f96cc06c0b0 0x7f96cc0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:53.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.062+0000 7f96caffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f96cc15a2e0 0x7f96cc06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48624/0 (socket says 172.21.15.129:48624) 2024-03-23T15:50:53.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.062+0000 7f96caffd640 1 -- 172.21.15.129:0/2111468993 learned_addr learned my addr 172.21.15.129:0/2111468993 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:53.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.066+0000 7f96cbfff640 1 -- 172.21.15.129:0/2111468993 >> v2:172.21.15.129:3300/0 conn(0x7f96cc15a2e0 msgr2=0x7f96cc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:53.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.066+0000 7f96cbfff640 1 --2- 172.21.15.129:0/2111468993 >> v2:172.21.15.129:3300/0 conn(0x7f96cc15a2e0 0x7f96cc06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.066+0000 7f96cbfff640 1 -- 172.21.15.129:0/2111468993 >> v1:172.21.15.129:6789/0 conn(0x7f96cc1544b0 legacy=0x7f96cc0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:53.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.066+0000 7f96cbfff640 1 -- 172.21.15.129:0/2111468993 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96cc15a9e0 con 0x7f96cc06c0b0 2024-03-23T15:50:53.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.069+0000 7f96cbfff640 1 --2- 172.21.15.129:0/2111468993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc06c0b0 0x7f96cc0c6000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0b3b50 tx=0x7f96c402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b97c3972af513fb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:53.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.074+0000 7f96ca7fc640 1 -- 172.21.15.129:0/2111468993 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c402cd00 con 0x7f96cc06c0b0 2024-03-23T15:50:53.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.074+0000 7f96ca7fc640 1 -- 172.21.15.129:0/2111468993 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c402ce60 con 0x7f96cc06c0b0 2024-03-23T15:50:53.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.074+0000 7f96ca7fc640 1 -- 172.21.15.129:0/2111468993 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c4034660 con 0x7f96cc06c0b0 2024-03-23T15:50:53.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 -- 172.21.15.129:0/2111468993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc06c0b0 msgr2=0x7f96cc0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:53.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/2111468993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc06c0b0 0x7f96cc0c6000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0b3b50 tx=0x7f96c402bf50 comp rx=0 tx=0).stop 2024-03-23T15:50:53.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 -- 172.21.15.129:0/2111468993 shutdown_connections 2024-03-23T15:50:53.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/2111468993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc06c0b0 0x7f96cc0c6000 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/2111468993 >> v2:172.21.15.129:3300/0 conn(0x7f96cc15a2e0 0x7f96cc06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 -- 172.21.15.129:0/2111468993 >> 172.21.15.129:0/2111468993 conn(0x7f96cc061b50 msgr2=0x7f96cc061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:53.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 -- 172.21.15.129:0/2111468993 shutdown_connections 2024-03-23T15:50:53.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.075+0000 7f96d1aeb640 1 -- 172.21.15.129:0/2111468993 wait complete. 2024-03-23T15:50:53.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.076+0000 7f96d1aeb640 1 Processor -- start 2024-03-23T15:50:53.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.076+0000 7f96d1aeb640 1 -- start start 2024-03-23T15:50:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.077+0000 7f96d1aeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 0x7f96cc1f1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.077+0000 7f96d1aeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96cc1544b0 0x7f96cc143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.077+0000 7f96d1aeb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc15a2e0 0x7f96cc1444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.077+0000 7f96d1aeb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc1f1690 con 0x7f96cc06c0b0 2024-03-23T15:50:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.077+0000 7f96d1aeb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc0d0340 con 0x7f96cc15a2e0 2024-03-23T15:50:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.077+0000 7f96d1aeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96cc0d0620 con 0x7f96cc1544b0 2024-03-23T15:50:53.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.078+0000 7f96cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 0x7f96cc1f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:53.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.078+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc15a2e0 0x7f96cc1444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:53.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.078+0000 7f96caffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96cc1544b0 0x7f96cc143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:53.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.078+0000 7f96cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 0x7f96cc1f1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48632/0 (socket says 172.21.15.129:48632) 2024-03-23T15:50:53.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.078+0000 7f96cb7fe640 1 -- 172.21.15.129:0/487383070 learned_addr learned my addr 172.21.15.129:0/487383070 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:53.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.081+0000 7f96cb7fe640 1 -- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96cc1544b0 msgr2=0x7f96cc143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:53.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.081+0000 7f96cb7fe640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96cc1544b0 0x7f96cc143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.081+0000 7f96cb7fe640 1 -- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc15a2e0 msgr2=0x7f96cc1444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:53.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.081+0000 7f96cb7fe640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc15a2e0 0x7f96cc1444f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.081+0000 7f96cb7fe640 1 -- 172.21.15.129:0/487383070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c4009a70 con 0x7f96cc06c0b0 2024-03-23T15:50:53.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.084+0000 7f96cb7fe640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 0x7f96cc1f1150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00c400 tx=0x7f96bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:53.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.089+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0052d0 con 0x7f96cc06c0b0 2024-03-23T15:50:53.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.089+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96bc005450 con 0x7f96cc06c0b0 2024-03-23T15:50:53.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.089+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc007260 con 0x7f96cc06c0b0 2024-03-23T15:50:53.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.089+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96cc142bf0 con 0x7f96cc06c0b0 2024-03-23T15:50:53.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.089+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96cc0ceaf0 con 0x7f96cc06c0b0 2024-03-23T15:50:53.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.090+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96cc153100 con 0x7f96cc06c0b0 2024-03-23T15:50:53.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.380+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f96bc0073c0 con 0x7f96cc06c0b0 2024-03-23T15:50:53.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.380+0000 7f96b3fff640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f96b806b810 0x7f96b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:53.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.380+0000 7f96caffd640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f96b806b810 0x7f96b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:53.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.381+0000 7f96caffd640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f96b806b810 0x7f96b806dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f96cc14afd0 tx=0x7f96c0009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:53.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.396+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f96bc007670 con 0x7f96cc06c0b0 2024-03-23T15:50:53.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.396+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96bc0548d0 con 0x7f96cc06c0b0 2024-03-23T15:50:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.662+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96cc0c6100 con 0x7f96cc06c0b0 2024-03-23T15:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.682+0000 7f96b3fff640 1 -- 172.21.15.129:0/487383070 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f96bc020070 con 0x7f96cc06c0b0 2024-03-23T15:50:53.686 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:53.686 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:53.686 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f96b806b810 msgr2=0x7f96b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f96b806b810 0x7f96b806dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f96cc14afd0 tx=0x7f96c0009240 comp rx=0 tx=0).stop 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 msgr2=0x7f96cc1f1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 0x7f96cc1f1150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00c400 tx=0x7f96bc00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 shutdown_connections 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f96b806b810 0x7f96b806dcd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96cc15a2e0 0x7f96cc1444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96cc1544b0 0x7f96cc143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 --2- 172.21.15.129:0/487383070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc06c0b0 0x7f96cc1f1150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 >> 172.21.15.129:0/487383070 conn(0x7f96cc061b50 msgr2=0x7f96cc0b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 shutdown_connections 2024-03-23T15:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:53.685+0000 7f96d1aeb640 1 -- 172.21.15.129:0/487383070 wait complete. 2024-03-23T15:50:54.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:54.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:55.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:55.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.404+0000 7f8f5842c640 1 Processor -- start 2024-03-23T15:50:55.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.404+0000 7f8f5842c640 1 -- start start 2024-03-23T15:50:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.405+0000 7f8f5842c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f5015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.405+0000 7f8f5842c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8f500cef40 0x7f8f5015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.405+0000 7f8f5842c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5015b290 con 0x7f8f500ceb00 2024-03-23T15:50:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.405+0000 7f8f5842c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8f500ce800 con 0x7f8f500cef40 2024-03-23T15:50:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.405+0000 7f8f5842c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f50154a70 con 0x7f8f500ccd60 2024-03-23T15:50:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.406+0000 7f8f569c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8f500cef40 0x7f8f5015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.406+0000 7f8f561c8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f5015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.406+0000 7f8f569c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8f500cef40 0x7f8f5015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48072/0 (socket says 172.21.15.129:48072) 2024-03-23T15:50:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.406+0000 7f8f569c9640 1 -- 172.21.15.129:0/1827279868 learned_addr learned my addr 172.21.15.129:0/1827279868 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.409+0000 7f8f569c9640 1 -- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 msgr2=0x7f8f5015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.409+0000 7f8f569c9640 1 --2- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f5015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.409+0000 7f8f569c9640 1 -- 172.21.15.129:0/1827279868 >> v1:172.21.15.157:6789/0 conn(0x7f8f500ceb00 legacy=0x7f8f5006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.409+0000 7f8f569c9640 1 -- 172.21.15.129:0/1827279868 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f500cd950 con 0x7f8f500cef40 2024-03-23T15:50:55.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.411+0000 7f8f569c9640 1 --2- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f5015b8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009e50 tx=0x7f8f4802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf5618377d4acbb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:55.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.417+0000 7f8f551c6640 1 -- 172.21.15.129:0/1827279868 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4802cb00 con 0x7f8f500cef40 2024-03-23T15:50:55.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.417+0000 7f8f551c6640 1 -- 172.21.15.129:0/1827279868 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f4802cc60 con 0x7f8f500cef40 2024-03-23T15:50:55.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f551c6640 1 -- 172.21.15.129:0/1827279868 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f480343f0 con 0x7f8f500cef40 2024-03-23T15:50:55.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 -- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 msgr2=0x7f8f5015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:55.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 --2- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f5015b8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009e50 tx=0x7f8f4802bf60 comp rx=0 tx=0).stop 2024-03-23T15:50:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 -- 172.21.15.129:0/1827279868 shutdown_connections 2024-03-23T15:50:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 --2- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f5015b8f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 --2- 172.21.15.129:0/1827279868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f5015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 -- 172.21.15.129:0/1827279868 >> 172.21.15.129:0/1827279868 conn(0x7f8f500b1840 msgr2=0x7f8f500b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 -- 172.21.15.129:0/1827279868 shutdown_connections 2024-03-23T15:50:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.418+0000 7f8f5842c640 1 -- 172.21.15.129:0/1827279868 wait complete. 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.419+0000 7f8f5842c640 1 Processor -- start 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.419+0000 7f8f5842c640 1 -- start start 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.420+0000 7f8f5842c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f501f0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.420+0000 7f8f5842c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f500ceb00 0x7f8f501e53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.420+0000 7f8f5842c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f50152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.420+0000 7f8f5842c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f501e3ba0 con 0x7f8f500ceb00 2024-03-23T15:50:55.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.420+0000 7f8f5842c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8f501f0390 con 0x7f8f500cef40 2024-03-23T15:50:55.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.420+0000 7f8f5842c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f501f0910 con 0x7f8f500ccd60 2024-03-23T15:50:55.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.421+0000 7f8f569c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f50152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:55.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.421+0000 7f8f559c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f500ceb00 0x7f8f501e53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:55.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.421+0000 7f8f569c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f50152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48088/0 (socket says 172.21.15.129:48088) 2024-03-23T15:50:55.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.421+0000 7f8f569c9640 1 -- 172.21.15.129:0/3961215190 learned_addr learned my addr 172.21.15.129:0/3961215190 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:55.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.421+0000 7f8f561c8640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f501f0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:55.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.425+0000 7f8f559c7640 1 -- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 msgr2=0x7f8f501f0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:55.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.425+0000 7f8f559c7640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f501f0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.425+0000 7f8f559c7640 1 -- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 msgr2=0x7f8f50152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:55.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.425+0000 7f8f559c7640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f50152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.425+0000 7f8f559c7640 1 -- 172.21.15.129:0/3961215190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f48009b30 con 0x7f8f500ceb00 2024-03-23T15:50:55.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.425+0000 7f8f569c9640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f50152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:50:55.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.427+0000 7f8f559c7640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f500ceb00 0x7f8f501e53a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000b3e0 tx=0x7f8f4000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:55.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.431+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40011830 con 0x7f8f500ceb00 2024-03-23T15:50:55.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.431+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f40011990 con 0x7f8f500ceb00 2024-03-23T15:50:55.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.431+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f501f12e0 con 0x7f8f500ceb00 2024-03-23T15:50:55.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.431+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40011d40 con 0x7f8f500ceb00 2024-03-23T15:50:55.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.431+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f501e38c0 con 0x7f8f500ceb00 2024-03-23T15:50:55.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.433+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5004f530 con 0x7f8f500ceb00 2024-03-23T15:50:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.687+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8f4001a400 con 0x7f8f500ceb00 2024-03-23T15:50:55.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.687+0000 7f8f46ffd640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f3406b860 0x7f8f3406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:55.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.687+0000 7f8f561c8640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f3406b860 0x7f8f3406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:55.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.688+0000 7f8f561c8640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f3406b860 0x7f8f3406dd20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009aa0 tx=0x7f8f4c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:55.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.700+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8f40059020 con 0x7f8f500ceb00 2024-03-23T15:50:55.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.700+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8f400552c0 con 0x7f8f500ceb00 2024-03-23T15:50:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.942+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f5006db50 con 0x7f8f500ceb00 2024-03-23T15:50:55.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.957+0000 7f8f46ffd640 1 -- 172.21.15.129:0/3961215190 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8f40015090 con 0x7f8f500ceb00 2024-03-23T15:50:55.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:55.962 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f3406b860 msgr2=0x7f8f3406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f3406b860 0x7f8f3406dd20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009aa0 tx=0x7f8f4c009190 comp rx=0 tx=0).stop 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f500ceb00 msgr2=0x7f8f501e53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f500ceb00 0x7f8f501e53a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000b3e0 tx=0x7f8f4000b8a0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 shutdown_connections 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f3406b860 0x7f8f3406dd20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f500cef40 0x7f8f50152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f500ceb00 0x7f8f501e53a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 --2- 172.21.15.129:0/3961215190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f500ccd60 0x7f8f501f0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 >> 172.21.15.129:0/3961215190 conn(0x7f8f500b1840 msgr2=0x7f8f500b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:55.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 shutdown_connections 2024-03-23T15:50:55.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:55.959+0000 7f8f5842c640 1 -- 172.21.15.129:0/3961215190 wait complete. 2024-03-23T15:50:56.578 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:56.578 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:57.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:57.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.679+0000 7fd2f3513640 1 Processor -- start 2024-03-23T15:50:57.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.679+0000 7fd2f3513640 1 -- start start 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.680+0000 7fd2f3513640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd2ec1544b0 0x7fd2ec15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.680+0000 7fd2f3513640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec0caa00 0x7fd2ec06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.680+0000 7fd2f3513640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec0cbcb0 con 0x7fd2ec06c0b0 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.680+0000 7fd2f3513640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd2ec0c5be0 con 0x7fd2ec1544b0 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.680+0000 7fd2f3513640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec15b280 con 0x7fd2ec0caa00 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.680+0000 7fd2f0aae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec0caa00 0x7fd2ec06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.681+0000 7fd2f12af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd2ec1544b0 0x7fd2ec15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.681+0000 7fd2f0aae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec0caa00 0x7fd2ec06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:56824/0 (socket says 172.21.15.129:56824) 2024-03-23T15:50:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.681+0000 7fd2f0aae640 1 -- 172.21.15.129:0/1818184879 learned_addr learned my addr 172.21.15.129:0/1818184879 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:57.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.684+0000 7fd2f12af640 1 -- 172.21.15.129:0/1818184879 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec0caa00 msgr2=0x7fd2ec06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:57.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.684+0000 7fd2f12af640 1 --2- 172.21.15.129:0/1818184879 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec0caa00 0x7fd2ec06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:57.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.684+0000 7fd2f12af640 1 -- 172.21.15.129:0/1818184879 >> v1:172.21.15.129:6789/0 conn(0x7fd2ec06c0b0 legacy=0x7fd2ec0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:57.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.684+0000 7fd2f12af640 1 -- 172.21.15.129:0/1818184879 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec15a9e0 con 0x7fd2ec1544b0 2024-03-23T15:50:57.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.688+0000 7fd2f12af640 1 --2- 172.21.15.129:0/1818184879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec1544b0 0x7fd2ec15a2e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80097a0 tx=0x7fd2e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9d44d21fb0bd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:57.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.693+0000 7fd2e3fff640 1 -- 172.21.15.129:0/1818184879 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8030610 con 0x7fd2ec1544b0 2024-03-23T15:50:57.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.693+0000 7fd2e3fff640 1 -- 172.21.15.129:0/1818184879 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e8030770 con 0x7fd2ec1544b0 2024-03-23T15:50:57.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.693+0000 7fd2e3fff640 1 -- 172.21.15.129:0/1818184879 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8030b00 con 0x7fd2ec1544b0 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 -- 172.21.15.129:0/1818184879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec1544b0 msgr2=0x7fd2ec15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1818184879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec1544b0 0x7fd2ec15a2e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80097a0 tx=0x7fd2e802b960 comp rx=0 tx=0).stop 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 -- 172.21.15.129:0/1818184879 shutdown_connections 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1818184879 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec0caa00 0x7fd2ec06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1818184879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec1544b0 0x7fd2ec15a2e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 -- 172.21.15.129:0/1818184879 >> 172.21.15.129:0/1818184879 conn(0x7fd2ec061b50 msgr2=0x7fd2ec061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:57.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 -- 172.21.15.129:0/1818184879 shutdown_connections 2024-03-23T15:50:57.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.694+0000 7fd2f3513640 1 -- 172.21.15.129:0/1818184879 wait complete. 2024-03-23T15:50:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.696+0000 7fd2f3513640 1 Processor -- start 2024-03-23T15:50:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.696+0000 7fd2f3513640 1 -- start start 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f3513640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2ec06c0b0 0x7fd2ec148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f3513640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 0x7fd2ec142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f3513640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec1544b0 0x7fd2ec1415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f3513640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec1563e0 con 0x7fd2ec0caa00 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f3513640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec0d0390 con 0x7fd2ec06c0b0 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f3513640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec157970 con 0x7fd2ec1544b0 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.697+0000 7fd2f0aae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 0x7fd2ec142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.698+0000 7fd2f1ab0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec1544b0 0x7fd2ec1415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.698+0000 7fd2f0aae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 0x7fd2ec142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48666/0 (socket says 172.21.15.129:48666) 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.698+0000 7fd2f0aae640 1 -- 172.21.15.129:0/1564351630 learned_addr learned my addr 172.21.15.129:0/1564351630 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:57.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.698+0000 7fd2f12af640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2ec06c0b0 0x7fd2ec148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:57.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.701+0000 7fd2f0aae640 1 -- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec1544b0 msgr2=0x7fd2ec1415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:57.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.701+0000 7fd2f0aae640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec1544b0 0x7fd2ec1415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:57.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.701+0000 7fd2f0aae640 1 -- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2ec06c0b0 msgr2=0x7fd2ec148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:57.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.701+0000 7fd2f0aae640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2ec06c0b0 0x7fd2ec148540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:57.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.701+0000 7fd2f0aae640 1 -- 172.21.15.129:0/1564351630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e8009400 con 0x7fd2ec0caa00 2024-03-23T15:50:57.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.704+0000 7fd2f0aae640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 0x7fd2ec142690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009fd0 tx=0x7fd2dc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:57.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.708+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00c940 con 0x7fd2ec0caa00 2024-03-23T15:50:57.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.708+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2dc00caa0 con 0x7fd2ec0caa00 2024-03-23T15:50:57.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.708+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00ce30 con 0x7fd2ec0caa00 2024-03-23T15:50:57.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.708+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ec0d05a0 con 0x7fd2ec0caa00 2024-03-23T15:50:57.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.708+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ec155960 con 0x7fd2ec0caa00 2024-03-23T15:50:57.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.709+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ec0ca770 con 0x7fd2ec0caa00 2024-03-23T15:50:57.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.738+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2dc013470 con 0x7fd2ec0caa00 2024-03-23T15:50:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.739+0000 7fd2e1ffb640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2d006b860 0x7fd2d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.739+0000 7fd2f12af640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2d006b860 0x7fd2d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.740+0000 7fd2f12af640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2d006b860 0x7fd2d006dd20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8036000 tx=0x7fd2e803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:57.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.745+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd2dc014e40 con 0x7fd2ec0caa00 2024-03-23T15:50:57.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:57.996+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2dc055570 con 0x7fd2ec0caa00 2024-03-23T15:50:58.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.216+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2ec075740 con 0x7fd2ec0caa00 2024-03-23T15:50:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.229+0000 7fd2e1ffb640 1 -- 172.21.15.129:0/1564351630 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd2dc017090 con 0x7fd2ec0caa00 2024-03-23T15:50:58.232 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:50:58.232 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:50:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2d006b860 msgr2=0x7fd2d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2d006b860 0x7fd2d006dd20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8036000 tx=0x7fd2e803b040 comp rx=0 tx=0).stop 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 msgr2=0x7fd2ec142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 0x7fd2ec142690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009fd0 tx=0x7fd2dc00ed40 comp rx=0 tx=0).stop 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 shutdown_connections 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2ec1544b0 0x7fd2ec1415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec0caa00 0x7fd2ec142690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd2d006b860 0x7fd2d006dd20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 --2- 172.21.15.129:0/1564351630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2ec06c0b0 0x7fd2ec148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.231+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 >> 172.21.15.129:0/1564351630 conn(0x7fd2ec061b50 msgr2=0x7fd2ec14a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.232+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 shutdown_connections 2024-03-23T15:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:58.232+0000 7fd2f3513640 1 -- 172.21.15.129:0/1564351630 wait complete. 2024-03-23T15:50:58.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:50:58.850 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:50:59.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:50:59.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.951+0000 7f6efed63640 1 Processor -- start 2024-03-23T15:50:59.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.951+0000 7f6efed63640 1 -- start start 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efed63640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ef80ccd60 0x7f6ef815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efed63640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ef80cef40 0x7f6ef815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efed63640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef815b290 con 0x7f6ef80ceb00 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efed63640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef80ce800 con 0x7f6ef80cef40 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efed63640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef8154a70 con 0x7f6ef80ccd60 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efd300640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ef80cef40 0x7f6ef815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.952+0000 7f6efcaff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ef80ccd60 0x7f6ef815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:59.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.953+0000 7f6efcaff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ef80ccd60 0x7f6ef815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48106/0 (socket says 172.21.15.129:48106) 2024-03-23T15:50:59.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.953+0000 7f6efcaff640 1 -- 172.21.15.129:0/3983461028 learned_addr learned my addr 172.21.15.129:0/3983461028 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:59.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.955+0000 7f6efcaff640 1 -- 172.21.15.129:0/3983461028 >> v2:172.21.15.129:3300/0 conn(0x7f6ef80cef40 msgr2=0x7f6ef815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:59.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.955+0000 7f6efcaff640 1 --2- 172.21.15.129:0/3983461028 >> v2:172.21.15.129:3300/0 conn(0x7f6ef80cef40 0x7f6ef815b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:59.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.955+0000 7f6efcaff640 1 -- 172.21.15.129:0/3983461028 >> v1:172.21.15.129:6789/0 conn(0x7f6ef80ceb00 legacy=0x7f6ef806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:50:59.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.955+0000 7f6efcaff640 1 -- 172.21.15.129:0/3983461028 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef80cd950 con 0x7f6ef80ccd60 2024-03-23T15:50:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.957+0000 7f6efcaff640 1 --2- 172.21.15.129:0/3983461028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ccd60 0x7f6ef815ad50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ef808cf40 tx=0x7f6eec02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61e1e038e109667a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:59.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.960+0000 7f6ef77fe640 1 -- 172.21.15.129:0/3983461028 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec030860 con 0x7f6ef80ccd60 2024-03-23T15:50:59.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.962+0000 7f6ef77fe640 1 -- 172.21.15.129:0/3983461028 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eec0309c0 con 0x7f6ef80ccd60 2024-03-23T15:50:59.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.962+0000 7f6ef77fe640 1 -- 172.21.15.129:0/3983461028 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec030d50 con 0x7f6ef80ccd60 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 -- 172.21.15.129:0/3983461028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ccd60 msgr2=0x7f6ef815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 --2- 172.21.15.129:0/3983461028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ccd60 0x7f6ef815ad50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ef808cf40 tx=0x7f6eec02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 -- 172.21.15.129:0/3983461028 shutdown_connections 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 --2- 172.21.15.129:0/3983461028 >> v2:172.21.15.129:3300/0 conn(0x7f6ef80cef40 0x7f6ef815b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 --2- 172.21.15.129:0/3983461028 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ccd60 0x7f6ef815ad50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 -- 172.21.15.129:0/3983461028 >> 172.21.15.129:0/3983461028 conn(0x7f6ef80b1840 msgr2=0x7f6ef80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 -- 172.21.15.129:0/3983461028 shutdown_connections 2024-03-23T15:50:59.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.963+0000 7f6efed63640 1 -- 172.21.15.129:0/3983461028 wait complete. 2024-03-23T15:50:59.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.964+0000 7f6efed63640 1 Processor -- start 2024-03-23T15:50:59.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.964+0000 7f6efed63640 1 -- start start 2024-03-23T15:50:59.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6efed63640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ef80ccd60 0x7f6ef81f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6efed63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ceb00 0x7f6ef80c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6efed63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ef80cef40 0x7f6ef8151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6efed63640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef80c3b20 con 0x7f6ef80ccd60 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6efed63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef81f1350 con 0x7f6ef80ceb00 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6efed63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef80c5620 con 0x7f6ef80cef40 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.965+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ceb00 0x7f6ef80c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.966+0000 7f6efcaff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ef80ccd60 0x7f6ef81f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.966+0000 7f6efd300640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ef80cef40 0x7f6ef8151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:59.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.966+0000 7f6ef7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ceb00 0x7f6ef80c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48116/0 (socket says 172.21.15.129:48116) 2024-03-23T15:50:59.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.966+0000 7f6ef7fff640 1 -- 172.21.15.129:0/2580501684 learned_addr learned my addr 172.21.15.129:0/2580501684 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:50:59.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.969+0000 7f6efcaff640 1 -- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ef80cef40 msgr2=0x7f6ef8151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:59.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.969+0000 7f6efcaff640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ef80cef40 0x7f6ef8151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:59.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.969+0000 7f6efcaff640 1 -- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ceb00 msgr2=0x7f6ef80c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:50:59.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.969+0000 7f6efcaff640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ceb00 0x7f6ef80c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:50:59.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.969+0000 7f6efcaff640 1 -- 172.21.15.129:0/2580501684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec0094d0 con 0x7f6ef80ccd60 2024-03-23T15:50:59.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.972+0000 7f6efcaff640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ef80ccd60 0x7f6ef81f0e10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6eec033000 tx=0x7f6eec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:50:59.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.975+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec039a90 con 0x7f6ef80ccd60 2024-03-23T15:50:59.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.975+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eec039bf0 con 0x7f6ef80ccd60 2024-03-23T15:50:59.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.975+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0433f0 con 0x7f6ef80ccd60 2024-03-23T15:50:59.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.975+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef80c4770 con 0x7f6ef80ccd60 2024-03-23T15:50:59.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.975+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef80c4cf0 con 0x7f6ef80ccd60 2024-03-23T15:50:59.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.977+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef804f530 con 0x7f6ef80ccd60 2024-03-23T15:50:59.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.995+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6eec04c3b0 con 0x7f6ef80ccd60 2024-03-23T15:50:59.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.995+0000 7f6ef57fa640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6edc06b860 0x7f6edc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:50:59.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.996+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6edc06b860 0x7f6edc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:50:59.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.996+0000 7f6ef7fff640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6edc06b860 0x7f6edc06dd20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8068540 tx=0x7f6ee4009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:00.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:50:59.999+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6eec07b5e0 con 0x7f6ef80ccd60 2024-03-23T15:51:00.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.255+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6eec077880 con 0x7f6ef80ccd60 2024-03-23T15:51:00.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.490+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ef806db50 con 0x7f6ef80ccd60 2024-03-23T15:51:00.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.551+0000 7f6ef57fa640 1 -- 172.21.15.129:0/2580501684 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6eec034020 con 0x7f6ef80ccd60 2024-03-23T15:51:00.554 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:00.554 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:00.555 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:00.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6edc06b860 msgr2=0x7f6edc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:00.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6edc06b860 0x7f6edc06dd20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8068540 tx=0x7f6ee4009110 comp rx=0 tx=0).stop 2024-03-23T15:51:00.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ef80ccd60 msgr2=0x7f6ef81f0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:00.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ef80ccd60 0x7f6ef81f0e10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6eec033000 tx=0x7f6eec004730 comp rx=0 tx=0).stop 2024-03-23T15:51:00.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 shutdown_connections 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ef80cef40 0x7f6ef8151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6edc06b860 0x7f6edc06dd20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ef80ceb00 0x7f6ef80c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 --2- 172.21.15.129:0/2580501684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ef80ccd60 0x7f6ef81f0e10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 >> 172.21.15.129:0/2580501684 conn(0x7f6ef80b1840 msgr2=0x7f6ef8153fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 shutdown_connections 2024-03-23T15:51:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:00.553+0000 7f6efed63640 1 -- 172.21.15.129:0/2580501684 wait complete. 2024-03-23T15:51:01.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:01.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:02.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:02.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.280+0000 7f5288f9a640 1 Processor -- start 2024-03-23T15:51:02.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.280+0000 7f5288f9a640 1 -- start start 2024-03-23T15:51:02.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.281+0000 7f5288f9a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f52840c8e90 0x7f52840c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:02.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.281+0000 7f5288f9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 0x7f5284157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.281+0000 7f5288f9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52840c3bd0 con 0x7f52840c7380 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.281+0000 7f5288f9a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f52840c57d0 con 0x7f52840c8e90 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.281+0000 7f5288f9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f52840c7ca0 con 0x7f5284156f60 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.282+0000 7f5281d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f52840c8e90 0x7f52840c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.282+0000 7f5282d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 0x7f5284157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.282+0000 7f5282d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 0x7f5284157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44058/0 (socket says 172.21.15.129:44058) 2024-03-23T15:51:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.282+0000 7f5282d9d640 1 -- 172.21.15.129:0/3217700881 learned_addr learned my addr 172.21.15.129:0/3217700881 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:02.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.285+0000 7f5282d9d640 1 -- 172.21.15.129:0/3217700881 >> v2:172.21.15.129:3300/0 conn(0x7f52840c8e90 msgr2=0x7f52840c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:02.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.285+0000 7f5282d9d640 1 --2- 172.21.15.129:0/3217700881 >> v2:172.21.15.129:3300/0 conn(0x7f52840c8e90 0x7f52840c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.285+0000 7f5282d9d640 1 -- 172.21.15.129:0/3217700881 >> v1:172.21.15.157:6789/0 conn(0x7f52840c7380 legacy=0x7f52840c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:02.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.285+0000 7f5282d9d640 1 -- 172.21.15.129:0/3217700881 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52840c3740 con 0x7f5284156f60 2024-03-23T15:51:02.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.287+0000 7f5282d9d640 1 --2- 172.21.15.129:0/3217700881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 0x7f5284157390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f527c009e50 tx=0x7f527c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3904697dbd42b635 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:02.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.292+0000 7f528159a640 1 -- 172.21.15.129:0/3217700881 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c02cb00 con 0x7f5284156f60 2024-03-23T15:51:02.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.292+0000 7f528159a640 1 -- 172.21.15.129:0/3217700881 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527c02cc60 con 0x7f5284156f60 2024-03-23T15:51:02.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.292+0000 7f528159a640 1 -- 172.21.15.129:0/3217700881 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c0343f0 con 0x7f5284156f60 2024-03-23T15:51:02.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 -- 172.21.15.129:0/3217700881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 msgr2=0x7f5284157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:02.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 --2- 172.21.15.129:0/3217700881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 0x7f5284157390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f527c009e50 tx=0x7f527c02bf60 comp rx=0 tx=0).stop 2024-03-23T15:51:02.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 -- 172.21.15.129:0/3217700881 shutdown_connections 2024-03-23T15:51:02.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 --2- 172.21.15.129:0/3217700881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5284156f60 0x7f5284157390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 --2- 172.21.15.129:0/3217700881 >> v2:172.21.15.129:3300/0 conn(0x7f52840c8e90 0x7f52840c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 -- 172.21.15.129:0/3217700881 >> 172.21.15.129:0/3217700881 conn(0x7f52840c8370 msgr2=0x7f52840c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:02.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 -- 172.21.15.129:0/3217700881 shutdown_connections 2024-03-23T15:51:02.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.293+0000 7f5288f9a640 1 -- 172.21.15.129:0/3217700881 wait complete. 2024-03-23T15:51:02.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.295+0000 7f5288f9a640 1 Processor -- start 2024-03-23T15:51:02.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.295+0000 7f5288f9a640 1 -- start start 2024-03-23T15:51:02.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.296+0000 7f5288f9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52840c7380 0x7f52841ece20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.296+0000 7f5288f9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 0x7f5284151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.296+0000 7f5288f9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284156f60 0x7f52841e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.296+0000 7f5288f9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5284142d70 con 0x7f5284156f60 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.296+0000 7f5288f9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f52841ee290 con 0x7f52840c7380 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.296+0000 7f5288f9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f52841ed410 con 0x7f52840c8e90 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.297+0000 7f5282d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284156f60 0x7f52841e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:02.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.297+0000 7f528259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52840c7380 0x7f52841ece20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:02.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.297+0000 7f5281d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 0x7f5284151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:02.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.297+0000 7f5281d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 0x7f5284151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44060/0 (socket says 172.21.15.129:44060) 2024-03-23T15:51:02.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.297+0000 7f5281d9b640 1 -- 172.21.15.129:0/987203387 learned_addr learned my addr 172.21.15.129:0/987203387 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:02.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.302+0000 7f5281d9b640 1 -- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52840c7380 msgr2=0x7f52841ece20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:02.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.302+0000 7f5281d9b640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52840c7380 0x7f52841ece20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.302+0000 7f5281d9b640 1 -- 172.21.15.129:0/987203387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284156f60 msgr2=0x7f52841e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:02.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.302+0000 7f5281d9b640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284156f60 0x7f52841e1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.302+0000 7f5281d9b640 1 -- 172.21.15.129:0/987203387 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527c009b30 con 0x7f52840c8e90 2024-03-23T15:51:02.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.302+0000 7f5282d9d640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284156f60 0x7f52841e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:51:02.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.306+0000 7f5281d9b640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 0x7f5284151130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f527800e9b0 tx=0x7f527800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:02.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.308+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278009670 con 0x7f52840c8e90 2024-03-23T15:51:02.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.308+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52841ed6d0 con 0x7f52840c8e90 2024-03-23T15:51:02.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.308+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52841edc90 con 0x7f52840c8e90 2024-03-23T15:51:02.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.309+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52841562c0 con 0x7f52840c8e90 2024-03-23T15:51:02.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.311+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527800c9a0 con 0x7f52840c8e90 2024-03-23T15:51:02.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.311+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527800cd10 con 0x7f52840c8e90 2024-03-23T15:51:02.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.628+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5278013470 con 0x7f52840c8e90 2024-03-23T15:51:02.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.628+0000 7f5272ffd640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f526406b810 0x7f526406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:02.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.629+0000 7f528259c640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f526406b810 0x7f526406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:02.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.629+0000 7f528259c640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f526406b810 0x7f526406dcd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5274000f60 tx=0x7f5274009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:02.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.633+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5278014e40 con 0x7f52840c8e90 2024-03-23T15:51:02.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.644+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5278055610 con 0x7f52840c8e90 2024-03-23T15:51:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.865+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5284075740 con 0x7f52840c8e90 2024-03-23T15:51:02.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.883+0000 7f5272ffd640 1 -- 172.21.15.129:0/987203387 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5278021090 con 0x7f52840c8e90 2024-03-23T15:51:02.887 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:02.887 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:02.887 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:02.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f526406b810 msgr2=0x7f526406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:02.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f526406b810 0x7f526406dcd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5274000f60 tx=0x7f5274009040 comp rx=0 tx=0).stop 2024-03-23T15:51:02.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 msgr2=0x7f5284151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:02.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 0x7f5284151130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f527800e9b0 tx=0x7f527800ee70 comp rx=0 tx=0).stop 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 shutdown_connections 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5284156f60 0x7f52841e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52840c8e90 0x7f5284151130 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f526406b810 0x7f526406dcd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 --2- 172.21.15.129:0/987203387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52840c7380 0x7f52841ece20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 >> 172.21.15.129:0/987203387 conn(0x7f52840c8370 msgr2=0x7f528406d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 shutdown_connections 2024-03-23T15:51:02.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:02.885+0000 7f5288f9a640 1 -- 172.21.15.129:0/987203387 wait complete. 2024-03-23T15:51:03.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:03.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:04.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.610+0000 7f78c8af4640 1 Processor -- start 2024-03-23T15:51:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.610+0000 7f78c8af4640 1 -- start start 2024-03-23T15:51:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.611+0000 7f78c8af4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f78c40ccd60 0x7f78c415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.611+0000 7f78c8af4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 0x7f78c415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.611+0000 7f78c8af4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c415b290 con 0x7f78c40ceb00 2024-03-23T15:51:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.611+0000 7f78c8af4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f78c40ce800 con 0x7f78c40ccd60 2024-03-23T15:51:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.611+0000 7f78c8af4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78c4154a70 con 0x7f78c40cef40 2024-03-23T15:51:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.612+0000 7f78c2d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 0x7f78c415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.612+0000 7f78c259c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f78c40ccd60 0x7f78c415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.612+0000 7f78c2d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 0x7f78c415b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44064/0 (socket says 172.21.15.129:44064) 2024-03-23T15:51:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.612+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1276451778 learned_addr learned my addr 172.21.15.129:0/1276451778 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.616+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1276451778 >> v2:172.21.15.157:3300/0 conn(0x7f78c40ccd60 msgr2=0x7f78c415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.616+0000 7f78c2d9d640 1 --2- 172.21.15.129:0/1276451778 >> v2:172.21.15.157:3300/0 conn(0x7f78c40ccd60 0x7f78c415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.616+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1276451778 >> v1:172.21.15.157:6789/0 conn(0x7f78c40ceb00 legacy=0x7f78c406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.616+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1276451778 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c40cd950 con 0x7f78c40cef40 2024-03-23T15:51:04.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.619+0000 7f78c2d9d640 1 --2- 172.21.15.129:0/1276451778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 0x7f78c415b8f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009e50 tx=0x7f78bc02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=882a699d10503599 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:04.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.624+0000 7f78c159a640 1 -- 172.21.15.129:0/1276451778 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc02cb90 con 0x7f78c40cef40 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.624+0000 7f78c159a640 1 -- 172.21.15.129:0/1276451778 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78bc02ccf0 con 0x7f78c40cef40 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.624+0000 7f78c159a640 1 -- 172.21.15.129:0/1276451778 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc034610 con 0x7f78c40cef40 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 -- 172.21.15.129:0/1276451778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 msgr2=0x7f78c415b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1276451778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 0x7f78c415b8f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009e50 tx=0x7f78bc02bff0 comp rx=0 tx=0).stop 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 -- 172.21.15.129:0/1276451778 shutdown_connections 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1276451778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40cef40 0x7f78c415b8f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1276451778 >> v2:172.21.15.157:3300/0 conn(0x7f78c40ccd60 0x7f78c415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 -- 172.21.15.129:0/1276451778 >> 172.21.15.129:0/1276451778 conn(0x7f78c40b1840 msgr2=0x7f78c40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 -- 172.21.15.129:0/1276451778 shutdown_connections 2024-03-23T15:51:04.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.625+0000 7f78c8af4640 1 -- 172.21.15.129:0/1276451778 wait complete. 2024-03-23T15:51:04.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.627+0000 7f78c8af4640 1 Processor -- start 2024-03-23T15:51:04.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.627+0000 7f78c8af4640 1 -- start start 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.628+0000 7f78c8af4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40ccd60 0x7f78c40b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.628+0000 7f78c8af4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 0x7f78c41e3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.628+0000 7f78c8af4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 0x7f78c41e4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.628+0000 7f78c8af4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78c41e3970 con 0x7f78c40ceb00 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.628+0000 7f78c8af4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f78c4156900 con 0x7f78c40cef40 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.628+0000 7f78c8af4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78c4154f80 con 0x7f78c40ccd60 2024-03-23T15:51:04.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.629+0000 7f78c1d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 0x7f78c41e3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:04.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.629+0000 7f78c2d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 0x7f78c41e4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:04.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.629+0000 7f78c259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40ccd60 0x7f78c40b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:04.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.629+0000 7f78c1d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 0x7f78c41e3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44792/0 (socket says 172.21.15.129:44792) 2024-03-23T15:51:04.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.629+0000 7f78c1d9b640 1 -- 172.21.15.129:0/1536088739 learned_addr learned my addr 172.21.15.129:0/1536088739 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:04.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.629+0000 7f78c2d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 0x7f78c41e4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:53366/0 (socket says 172.21.15.129:53366) 2024-03-23T15:51:04.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.633+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40ccd60 msgr2=0x7f78c40b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:04.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.633+0000 7f78c2d9d640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40ccd60 0x7f78c40b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:04.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.633+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1536088739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 msgr2=0x7f78c41e3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:04.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.633+0000 7f78c2d9d640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 0x7f78c41e3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:04.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.633+0000 7f78c2d9d640 1 -- 172.21.15.129:0/1536088739 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc009b30 con 0x7f78c40cef40 2024-03-23T15:51:04.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.633+0000 7f78c1d9b640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 0x7f78c41e3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:51:04.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.636+0000 7f78c2d9d640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 0x7f78c41e4340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78bc032000 tx=0x7f78bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:04.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.638+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc009510 con 0x7f78c40cef40 2024-03-23T15:51:04.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.638+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c4156b60 con 0x7f78c40cef40 2024-03-23T15:51:04.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.639+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c41e3600 con 0x7f78c40cef40 2024-03-23T15:51:04.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.639+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c404f530 con 0x7f78c40cef40 2024-03-23T15:51:04.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.928+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78bc03e450 con 0x7f78c40cef40 2024-03-23T15:51:04.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.928+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78bc03e7c0 con 0x7f78c40cef40 2024-03-23T15:51:04.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.929+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f78bc03e9e0 con 0x7f78c40cef40 2024-03-23T15:51:04.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.929+0000 7f78b2ffd640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78a406b810 0x7f78a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:04.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.929+0000 7f78c259c640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78a406b810 0x7f78a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:04.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.930+0000 7f78c259c640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78a406b810 0x7f78a406dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009b40 tx=0x7f78b4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.932+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f78bc07b7f0 con 0x7f78c40cef40 2024-03-23T15:51:04.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:04.944+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f78bc077a90 con 0x7f78c40cef40 2024-03-23T15:51:05.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.169+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78c406db50 con 0x7f78c40cef40 2024-03-23T15:51:05.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.182+0000 7f78b2ffd640 1 -- 172.21.15.129:0/1536088739 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f78bc043090 con 0x7f78c40cef40 2024-03-23T15:51:05.185 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:05.186 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78a406b810 msgr2=0x7f78a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78a406b810 0x7f78a406dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009b40 tx=0x7f78b4009190 comp rx=0 tx=0).stop 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 msgr2=0x7f78c41e4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 0x7f78c41e4340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78bc032000 tx=0x7f78bc004a40 comp rx=0 tx=0).stop 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 shutdown_connections 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78a406b810 0x7f78a406dcd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78c40cef40 0x7f78c41e4340 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78c40ceb00 0x7f78c41e3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 --2- 172.21.15.129:0/1536088739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78c40ccd60 0x7f78c40b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 >> 172.21.15.129:0/1536088739 conn(0x7f78c40b1840 msgr2=0x7f78c40b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 shutdown_connections 2024-03-23T15:51:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:05.184+0000 7f78c8af4640 1 -- 172.21.15.129:0/1536088739 wait complete. 2024-03-23T15:51:05.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:05.811 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:06.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:51:06.611 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:51:06.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:06.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.911+0000 7f44ff7c2640 1 Processor -- start 2024-03-23T15:51:06.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.911+0000 7f44ff7c2640 1 -- start start 2024-03-23T15:51:06.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.912+0000 7f44ff7c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 0x7f44f815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:06.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.912+0000 7f44ff7c2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f44f80cef40 0x7f44f815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:06.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.912+0000 7f44ff7c2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f815b290 con 0x7f44f80ccd60 2024-03-23T15:51:06.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.912+0000 7f44ff7c2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f44f80ce800 con 0x7f44f80cef40 2024-03-23T15:51:06.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.912+0000 7f44ff7c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f44f8154a70 con 0x7f44f80ceb00 2024-03-23T15:51:06.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.913+0000 7f44fdd5f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f44f80cef40 0x7f44f815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:06.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.913+0000 7f44fcd5d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 0x7f44f815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:06.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.913+0000 7f44fcd5d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 0x7f44f815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44070/0 (socket says 172.21.15.129:44070) 2024-03-23T15:51:06.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.913+0000 7f44fcd5d640 1 -- 172.21.15.129:0/3594008660 learned_addr learned my addr 172.21.15.129:0/3594008660 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:06.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.917+0000 7f44fcd5d640 1 -- 172.21.15.129:0/3594008660 >> v2:172.21.15.129:3300/0 conn(0x7f44f80cef40 msgr2=0x7f44f815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:06.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.917+0000 7f44fcd5d640 1 --2- 172.21.15.129:0/3594008660 >> v2:172.21.15.129:3300/0 conn(0x7f44f80cef40 0x7f44f815b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:06.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.917+0000 7f44fcd5d640 1 -- 172.21.15.129:0/3594008660 >> v1:172.21.15.129:6789/0 conn(0x7f44f80ccd60 legacy=0x7f44f806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:06.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.917+0000 7f44fcd5d640 1 -- 172.21.15.129:0/3594008660 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f80cd950 con 0x7f44f80ceb00 2024-03-23T15:51:06.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.920+0000 7f44fcd5d640 1 --2- 172.21.15.129:0/3594008660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 0x7f44f815ad50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009870 tx=0x7f44e802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4fed947fb06964 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:06.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.922+0000 7f44effff640 1 -- 172.21.15.129:0/3594008660 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e802f840 con 0x7f44f80ceb00 2024-03-23T15:51:06.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.924+0000 7f44effff640 1 -- 172.21.15.129:0/3594008660 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e802f9a0 con 0x7f44f80ceb00 2024-03-23T15:51:06.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.924+0000 7f44effff640 1 -- 172.21.15.129:0/3594008660 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e802fd30 con 0x7f44f80ceb00 2024-03-23T15:51:06.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 -- 172.21.15.129:0/3594008660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 msgr2=0x7f44f815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:06.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/3594008660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 0x7f44f815ad50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009870 tx=0x7f44e802ba20 comp rx=0 tx=0).stop 2024-03-23T15:51:06.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 -- 172.21.15.129:0/3594008660 shutdown_connections 2024-03-23T15:51:06.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/3594008660 >> v2:172.21.15.129:3300/0 conn(0x7f44f80cef40 0x7f44f815b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:06.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/3594008660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ceb00 0x7f44f815ad50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:06.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 -- 172.21.15.129:0/3594008660 >> 172.21.15.129:0/3594008660 conn(0x7f44f80b1840 msgr2=0x7f44f80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:06.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 -- 172.21.15.129:0/3594008660 shutdown_connections 2024-03-23T15:51:06.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.925+0000 7f44ff7c2640 1 -- 172.21.15.129:0/3594008660 wait complete. 2024-03-23T15:51:06.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.926+0000 7f44ff7c2640 1 Processor -- start 2024-03-23T15:51:06.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.926+0000 7f44ff7c2640 1 -- start start 2024-03-23T15:51:06.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44ff7c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ccd60 0x7f44f81f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:06.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44ff7c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 0x7f44f81f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:06.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44ff7c2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44f80cef40 0x7f44f81521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:06.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44ff7c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44f81509a0 con 0x7f44f80ceb00 2024-03-23T15:51:06.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44ff7c2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f44f81f50d0 con 0x7f44f80cef40 2024-03-23T15:51:06.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44ff7c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f44f8151750 con 0x7f44f80ccd60 2024-03-23T15:51:06.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.927+0000 7f44fcd5d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 0x7f44f81f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:06.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.928+0000 7f44fd55e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ccd60 0x7f44f81f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:06.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.928+0000 7f44fdd5f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44f80cef40 0x7f44f81521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:06.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.928+0000 7f44fcd5d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 0x7f44f81f15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44816/0 (socket says 172.21.15.129:44816) 2024-03-23T15:51:06.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.928+0000 7f44fcd5d640 1 -- 172.21.15.129:0/953001192 learned_addr learned my addr 172.21.15.129:0/953001192 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.931+0000 7f44fcd5d640 1 -- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ccd60 msgr2=0x7f44f81f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.931+0000 7f44fcd5d640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ccd60 0x7f44f81f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.931+0000 7f44fcd5d640 1 -- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44f80cef40 msgr2=0x7f44f81521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.931+0000 7f44fcd5d640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44f80cef40 0x7f44f81521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:06.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.931+0000 7f44fcd5d640 1 -- 172.21.15.129:0/953001192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e80094d0 con 0x7f44f80ceb00 2024-03-23T15:51:06.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.933+0000 7f44fcd5d640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 0x7f44f81f15b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.938+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8037c40 con 0x7f44f80ceb00 2024-03-23T15:51:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.938+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e8037da0 con 0x7f44f80ceb00 2024-03-23T15:51:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.938+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f8150b90 con 0x7f44f80ceb00 2024-03-23T15:51:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.938+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8041640 con 0x7f44f80ceb00 2024-03-23T15:51:06.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.938+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f8151470 con 0x7f44f80ceb00 2024-03-23T15:51:06.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:06.939+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f804f530 con 0x7f44f80ceb00 2024-03-23T15:51:07.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.207+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f44e80417a0 con 0x7f44f80ceb00 2024-03-23T15:51:07.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.207+0000 7f44edffb640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44dc06b810 0x7f44dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:07.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.207+0000 7f44fd55e640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44dc06b810 0x7f44dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:07.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.208+0000 7f44fd55e640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44dc06b810 0x7f44dc06dcd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f44f4001780 tx=0x7f44f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.232+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f44e803c070 con 0x7f44f80ceb00 2024-03-23T15:51:07.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.232+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44e8078590 con 0x7f44f80ceb00 2024-03-23T15:51:07.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.461+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44f80686f0 con 0x7f44f80ceb00 2024-03-23T15:51:07.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.486+0000 7f44edffb640 1 -- 172.21.15.129:0/953001192 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f44e807bc50 con 0x7f44f80ceb00 2024-03-23T15:51:07.489 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:07.489 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:07.490 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:07.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44dc06b810 msgr2=0x7f44dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:07.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44dc06b810 0x7f44dc06dcd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f44f4001780 tx=0x7f44f4009040 comp rx=0 tx=0).stop 2024-03-23T15:51:07.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 msgr2=0x7f44f81f15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 0x7f44f81f15b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44e8000c00 tx=0x7f44e8004730 comp rx=0 tx=0).stop 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 shutdown_connections 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44dc06b810 0x7f44dc06dcd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44f80cef40 0x7f44f81521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f80ceb00 0x7f44f81f15b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 --2- 172.21.15.129:0/953001192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44f80ccd60 0x7f44f81f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.488+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 >> 172.21.15.129:0/953001192 conn(0x7f44f80b1840 msgr2=0x7f44f8154230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.489+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 shutdown_connections 2024-03-23T15:51:07.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:07.489+0000 7f44ff7c2640 1 -- 172.21.15.129:0/953001192 wait complete. 2024-03-23T15:51:08.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:08.111 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:09.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.212+0000 7fea3359a640 1 Processor -- start 2024-03-23T15:51:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.212+0000 7fea3359a640 1 -- start start 2024-03-23T15:51:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.212+0000 7fea3359a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c15a2e0 0x7fea2c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea3359a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fea2c06c0b0 0x7fea2c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea3359a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea2c0cbcb0 con 0x7fea2c1544b0 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea3359a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fea2c0c5be0 con 0x7fea2c06c0b0 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea3359a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fea2c15b280 con 0x7fea2c15a2e0 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea31336640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fea2c1544b0 0x7fea2c0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:45700/0 (socket says 172.21.15.129:45700) 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea31336640 1 -- 172.21.15.129:0/4167186087 learned_addr learned my addr 172.21.15.129:0/4167186087 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea30b35640 1 --2- 172.21.15.129:0/4167186087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c15a2e0 0x7fea2c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:09.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.213+0000 7fea31b37640 1 --2- 172.21.15.129:0/4167186087 >> v2:172.21.15.157:3300/0 conn(0x7fea2c06c0b0 0x7fea2c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:09.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.216+0000 7fea23fff640 1 -- 172.21.15.129:0/4167186087 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070905325 0 0) 0x7fea2c0cbcb0 con 0x7fea2c1544b0 2024-03-23T15:51:09.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.216+0000 7fea23fff640 1 -- 172.21.15.129:0/4167186087 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea18003520 con 0x7fea2c1544b0 2024-03-23T15:51:09.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.217+0000 7fea30b35640 1 -- 172.21.15.129:0/4167186087 >> v2:172.21.15.157:3300/0 conn(0x7fea2c06c0b0 msgr2=0x7fea2c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:09.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.217+0000 7fea30b35640 1 --2- 172.21.15.129:0/4167186087 >> v2:172.21.15.157:3300/0 conn(0x7fea2c06c0b0 0x7fea2c0c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.217+0000 7fea30b35640 1 -- 172.21.15.129:0/4167186087 >> v1:172.21.15.129:6789/0 conn(0x7fea2c1544b0 legacy=0x7fea2c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:09.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.217+0000 7fea30b35640 1 -- 172.21.15.129:0/4167186087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c15a9e0 con 0x7fea2c15a2e0 2024-03-23T15:51:09.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.220+0000 7fea30b35640 1 --2- 172.21.15.129:0/4167186087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c15a2e0 0x7fea2c06bb70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0097a0 tx=0x7fea1c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26125de8679361a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.225+0000 7fea23fff640 1 -- 172.21.15.129:0/4167186087 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c030610 con 0x7fea2c15a2e0 2024-03-23T15:51:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.225+0000 7fea23fff640 1 -- 172.21.15.129:0/4167186087 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c030770 con 0x7fea2c15a2e0 2024-03-23T15:51:09.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.225+0000 7fea23fff640 1 -- 172.21.15.129:0/4167186087 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c030b00 con 0x7fea2c15a2e0 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 -- 172.21.15.129:0/4167186087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c15a2e0 msgr2=0x7fea2c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 --2- 172.21.15.129:0/4167186087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c15a2e0 0x7fea2c06bb70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0097a0 tx=0x7fea1c02b960 comp rx=0 tx=0).stop 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 -- 172.21.15.129:0/4167186087 shutdown_connections 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 --2- 172.21.15.129:0/4167186087 >> v2:172.21.15.157:3300/0 conn(0x7fea2c06c0b0 0x7fea2c0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 --2- 172.21.15.129:0/4167186087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c15a2e0 0x7fea2c06bb70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 -- 172.21.15.129:0/4167186087 >> 172.21.15.129:0/4167186087 conn(0x7fea2c061b50 msgr2=0x7fea2c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:09.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 -- 172.21.15.129:0/4167186087 shutdown_connections 2024-03-23T15:51:09.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.226+0000 7fea3359a640 1 -- 172.21.15.129:0/4167186087 wait complete. 2024-03-23T15:51:09.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.227+0000 7fea3359a640 1 Processor -- start 2024-03-23T15:51:09.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.227+0000 7fea3359a640 1 -- start start 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea3359a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea2c06c0b0 0x7fea2c154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea3359a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c1544b0 0x7fea2c1f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea3359a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea2c15a2e0 0x7fea2c1e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea3359a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c0c8230 con 0x7fea2c06c0b0 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea3359a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c155350 con 0x7fea2c15a2e0 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea3359a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fea2c1559f0 con 0x7fea2c1544b0 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea31336640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea2c06c0b0 0x7fea2c154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.228+0000 7fea30b35640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c1544b0 0x7fea2c1f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.229+0000 7fea31b37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea2c15a2e0 0x7fea2c1e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.229+0000 7fea30b35640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c1544b0 0x7fea2c1f0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44104/0 (socket says 172.21.15.129:44104) 2024-03-23T15:51:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.229+0000 7fea30b35640 1 -- 172.21.15.129:0/1342300793 learned_addr learned my addr 172.21.15.129:0/1342300793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:09.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.232+0000 7fea31336640 1 -- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c1544b0 msgr2=0x7fea2c1f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:09.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.232+0000 7fea31336640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c1544b0 0x7fea2c1f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.232+0000 7fea31336640 1 -- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea2c15a2e0 msgr2=0x7fea2c1e68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:09.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.232+0000 7fea31336640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea2c15a2e0 0x7fea2c1e68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.232+0000 7fea31336640 1 -- 172.21.15.129:0/1342300793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c009400 con 0x7fea2c06c0b0 2024-03-23T15:51:09.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.235+0000 7fea31336640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea2c06c0b0 0x7fea2c154e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c400 tx=0x7fea2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:09.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.240+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28005ce0 con 0x7fea2c06c0b0 2024-03-23T15:51:09.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.240+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea28005e40 con 0x7fea2c06c0b0 2024-03-23T15:51:09.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.240+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea280072f0 con 0x7fea2c06c0b0 2024-03-23T15:51:09.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.240+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c0c9160 con 0x7fea2c06c0b0 2024-03-23T15:51:09.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.241+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c0c7f50 con 0x7fea2c06c0b0 2024-03-23T15:51:09.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.242+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea2c075740 con 0x7fea2c06c0b0 2024-03-23T15:51:09.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.496+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea28007450 con 0x7fea2c06c0b0 2024-03-23T15:51:09.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.496+0000 7fea21ffb640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea1006b810 0x7fea1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:09.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.497+0000 7fea30b35640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea1006b810 0x7fea1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:09.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.497+0000 7fea30b35640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea1006b810 0x7fea1006dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fea2c1532b0 tx=0x7fea1c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:09.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.521+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fea28007e20 con 0x7fea2c06c0b0 2024-03-23T15:51:09.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.521+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea28054660 con 0x7fea2c06c0b0 2024-03-23T15:51:09.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.761+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea2c157d00 con 0x7fea2c06c0b0 2024-03-23T15:51:09.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.810+0000 7fea21ffb640 1 -- 172.21.15.129:0/1342300793 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fea28017030 con 0x7fea2c06c0b0 2024-03-23T15:51:09.813 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:09.813 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:09.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea1006b810 msgr2=0x7fea1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea1006b810 0x7fea1006dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fea2c1532b0 tx=0x7fea1c008030 comp rx=0 tx=0).stop 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea2c06c0b0 msgr2=0x7fea2c154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea2c06c0b0 0x7fea2c154e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c400 tx=0x7fea2800c8c0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 shutdown_connections 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea1006b810 0x7fea1006dcd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea2c15a2e0 0x7fea2c1e68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea2c1544b0 0x7fea2c1f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 --2- 172.21.15.129:0/1342300793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea2c06c0b0 0x7fea2c154e10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 >> 172.21.15.129:0/1342300793 conn(0x7fea2c061b50 msgr2=0x7fea2c0c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 shutdown_connections 2024-03-23T15:51:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:09.812+0000 7fea3359a640 1 -- 172.21.15.129:0/1342300793 wait complete. 2024-03-23T15:51:10.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:10.431 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:11.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.532+0000 7f3dea3ac640 1 Processor -- start 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.532+0000 7f3dea3ac640 1 -- start start 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3dea3ac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3de40ca580 0x7f3de406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3dea3ac640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de415d880 con 0x7f3de4154270 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3dea3ac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de40ca9d0 con 0x7f3de406c650 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3dea3ac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3de40c4ae0 con 0x7f3de40ca580 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3de3fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3de4154270 0x7f3de415ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:54136/0 (socket says 172.21.15.129:54136) 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3de3fff640 1 -- 172.21.15.129:0/1238056480 learned_addr learned my addr 172.21.15.129:0/1238056480 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.533+0000 7f3de37fe640 1 --2- 172.21.15.129:0/1238056480 >> v2:172.21.15.157:3300/0 conn(0x7f3de40ca580 0x7f3de406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.537+0000 7f3de2ffd640 1 -- 172.21.15.129:0/1238056480 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731241580 0 0) 0x7f3de415d880 con 0x7f3de4154270 2024-03-23T15:51:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.537+0000 7f3de2ffd640 1 -- 172.21.15.129:0/1238056480 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc003520 con 0x7f3de4154270 2024-03-23T15:51:11.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.538+0000 7f3de37fe640 1 -- 172.21.15.129:0/1238056480 >> v1:172.21.15.157:6789/0 conn(0x7f3de406c650 legacy=0x7f3de40c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:11.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.538+0000 7f3de37fe640 1 -- 172.21.15.129:0/1238056480 >> v1:172.21.15.129:6789/0 conn(0x7f3de4154270 legacy=0x7f3de415ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:11.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.538+0000 7f3de37fe640 1 -- 172.21.15.129:0/1238056480 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de40cd5e0 con 0x7f3de40ca580 2024-03-23T15:51:11.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.542+0000 7f3de37fe640 1 --2- 172.21.15.129:0/1238056480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de40ca580 0x7f3de406c110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80097a0 tx=0x7f3dd802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f68b0701e8c9db9b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.545+0000 7f3de2ffd640 1 -- 172.21.15.129:0/1238056480 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd802e070 con 0x7f3de40ca580 2024-03-23T15:51:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.546+0000 7f3de2ffd640 1 -- 172.21.15.129:0/1238056480 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd80315a0 con 0x7f3de40ca580 2024-03-23T15:51:11.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.546+0000 7f3de2ffd640 1 -- 172.21.15.129:0/1238056480 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8031910 con 0x7f3de40ca580 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 -- 172.21.15.129:0/1238056480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de40ca580 msgr2=0x7f3de406c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/1238056480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de40ca580 0x7f3de406c110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80097a0 tx=0x7f3dd802b840 comp rx=0 tx=0).stop 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 -- 172.21.15.129:0/1238056480 shutdown_connections 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/1238056480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de40ca580 0x7f3de406c110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 -- 172.21.15.129:0/1238056480 >> 172.21.15.129:0/1238056480 conn(0x7f3de406d290 msgr2=0x7f3de4072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 -- 172.21.15.129:0/1238056480 shutdown_connections 2024-03-23T15:51:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.547+0000 7f3dea3ac640 1 -- 172.21.15.129:0/1238056480 wait complete. 2024-03-23T15:51:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.548+0000 7f3dea3ac640 1 Processor -- start 2024-03-23T15:51:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.548+0000 7f3dea3ac640 1 -- start start 2024-03-23T15:51:11.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.549+0000 7f3dea3ac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de406c650 0x7f3de41e5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:11.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.549+0000 7f3dea3ac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de40ca580 0x7f3de414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:11.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.549+0000 7f3dea3ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3de4154270 0x7f3de414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:11.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.549+0000 7f3dea3ac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3de414d800 con 0x7f3de40ca580 2024-03-23T15:51:11.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.549+0000 7f3dea3ac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3de414eab0 con 0x7f3de406c650 2024-03-23T15:51:11.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.549+0000 7f3dea3ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3de414ef40 con 0x7f3de4154270 2024-03-23T15:51:11.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.550+0000 7f3de8949640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3de4154270 0x7f3de414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:11.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.550+0000 7f3de37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de40ca580 0x7f3de414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:11.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.550+0000 7f3de8949640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3de4154270 0x7f3de414d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51294/0 (socket says 172.21.15.129:51294) 2024-03-23T15:51:11.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.550+0000 7f3de8949640 1 -- 172.21.15.129:0/4149833437 learned_addr learned my addr 172.21.15.129:0/4149833437 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:11.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.553+0000 7f3de37fe640 1 -- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3de4154270 msgr2=0x7f3de414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:11.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.553+0000 7f3de37fe640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3de4154270 0x7f3de414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:11.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.553+0000 7f3de37fe640 1 -- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de406c650 msgr2=0x7f3de41e5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:11.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.553+0000 7f3de37fe640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de406c650 0x7f3de41e5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:11.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.553+0000 7f3de37fe640 1 -- 172.21.15.129:0/4149833437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd8009400 con 0x7f3de40ca580 2024-03-23T15:51:11.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.556+0000 7f3de37fe640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de40ca580 0x7f3de414cce0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8033000 tx=0x7f3dd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:11.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.558+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8003810 con 0x7f3de40ca580 2024-03-23T15:51:11.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.558+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de41f3f10 con 0x7f3de40ca580 2024-03-23T15:51:11.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.558+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de40cdc60 con 0x7f3de40ca580 2024-03-23T15:51:11.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.559+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de414e8a0 con 0x7f3de40ca580 2024-03-23T15:51:11.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.775+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd8040070 con 0x7f3de40ca580 2024-03-23T15:51:11.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.837+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8005b90 con 0x7f3de40ca580 2024-03-23T15:51:11.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.837+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3dd8003970 con 0x7f3de40ca580 2024-03-23T15:51:11.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.838+0000 7f3de0ff9640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3dd006b840 0x7f3dd006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:11.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.838+0000 7f3de3fff640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3dd006b840 0x7f3dd006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:11.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.839+0000 7f3de3fff640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3dd006b840 0x7f3dd006dd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4001570 tx=0x7f3dd4007930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:11.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.861+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3dd8042070 con 0x7f3de40ca580 2024-03-23T15:51:11.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:11.862+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3dd8077720 con 0x7f3de40ca580 2024-03-23T15:51:12.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.094+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3de414ec20 con 0x7f3de40ca580 2024-03-23T15:51:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.114+0000 7f3de0ff9640 1 -- 172.21.15.129:0/4149833437 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3dd8006e10 con 0x7f3de40ca580 2024-03-23T15:51:12.117 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:12.117 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:12.117 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3dd006b840 msgr2=0x7f3dd006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3dd006b840 0x7f3dd006dd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4001570 tx=0x7f3dd4007930 comp rx=0 tx=0).stop 2024-03-23T15:51:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de40ca580 msgr2=0x7f3de414cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de40ca580 0x7f3de414cce0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8033000 tx=0x7f3dd80046c0 comp rx=0 tx=0).stop 2024-03-23T15:51:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 shutdown_connections 2024-03-23T15:51:12.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3de4154270 0x7f3de414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de40ca580 0x7f3de414cce0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3dd006b840 0x7f3dd006dd00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 --2- 172.21.15.129:0/4149833437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3de406c650 0x7f3de41e5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 >> 172.21.15.129:0/4149833437 conn(0x7f3de406d290 msgr2=0x7f3de40cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 shutdown_connections 2024-03-23T15:51:12.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:12.117+0000 7f3dea3ac640 1 -- 172.21.15.129:0/4149833437 wait complete. 2024-03-23T15:51:12.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:12.736 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:13.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:13.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 Processor -- start 2024-03-23T15:51:13.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 -- start start 2024-03-23T15:51:13.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f80f815dc60 0x7f80f806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:13.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 0x7f80f80cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:13.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f806d2c0 con 0x7f80f806f570 2024-03-23T15:51:13.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f80f815e740 con 0x7f80f815dc60 2024-03-23T15:51:13.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.836+0000 7f80fd0e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f80f80ce2e0 con 0x7f80f815e040 2024-03-23T15:51:13.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.837+0000 7f80f659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 0x7f80f80cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:13.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.837+0000 7f80f6d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f80f815dc60 0x7f80f806cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:13.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.837+0000 7f80f659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 0x7f80f80cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51296/0 (socket says 172.21.15.129:51296) 2024-03-23T15:51:13.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.837+0000 7f80f659c640 1 -- 172.21.15.129:0/3828091821 learned_addr learned my addr 172.21.15.129:0/3828091821 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:13.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.840+0000 7f80f659c640 1 -- 172.21.15.129:0/3828091821 >> v2:172.21.15.157:3300/0 conn(0x7f80f815dc60 msgr2=0x7f80f806cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:13.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.840+0000 7f80f659c640 1 --2- 172.21.15.129:0/3828091821 >> v2:172.21.15.157:3300/0 conn(0x7f80f815dc60 0x7f80f806cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:13.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.840+0000 7f80f659c640 1 -- 172.21.15.129:0/3828091821 >> v1:172.21.15.157:6789/0 conn(0x7f80f806f570 legacy=0x7f80f80cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:13.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.840+0000 7f80f659c640 1 -- 172.21.15.129:0/3828091821 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f80ccd00 con 0x7f80f815e040 2024-03-23T15:51:13.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.842+0000 7f80f659c640 1 --2- 172.21.15.129:0/3828091821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 0x7f80f80cdda0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009870 tx=0x7f80ec02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceacbe6e23674ad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:13.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.846+0000 7f80f5d9b640 1 -- 172.21.15.129:0/3828091821 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec031640 con 0x7f80f815e040 2024-03-23T15:51:13.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.846+0000 7f80f5d9b640 1 -- 172.21.15.129:0/3828091821 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ec0317a0 con 0x7f80f815e040 2024-03-23T15:51:13.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.846+0000 7f80f5d9b640 1 -- 172.21.15.129:0/3828091821 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec031b30 con 0x7f80f815e040 2024-03-23T15:51:13.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 -- 172.21.15.129:0/3828091821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 msgr2=0x7f80f80cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/3828091821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 0x7f80f80cdda0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009870 tx=0x7f80ec02b8f0 comp rx=0 tx=0).stop 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 -- 172.21.15.129:0/3828091821 shutdown_connections 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/3828091821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815e040 0x7f80f80cdda0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/3828091821 >> v2:172.21.15.157:3300/0 conn(0x7f80f815dc60 0x7f80f806cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 -- 172.21.15.129:0/3828091821 >> 172.21.15.129:0/3828091821 conn(0x7f80f8071300 msgr2=0x7f80f80716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 -- 172.21.15.129:0/3828091821 shutdown_connections 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.847+0000 7f80fd0e8640 1 -- 172.21.15.129:0/3828091821 wait complete. 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.848+0000 7f80fd0e8640 1 Processor -- start 2024-03-23T15:51:13.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.848+0000 7f80fd0e8640 1 -- start start 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.849+0000 7f80fd0e8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80f806f570 0x7f80f81e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.849+0000 7f80fd0e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 0x7f80f8153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.849+0000 7f80fd0e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80f815e040 0x7f80f80c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.849+0000 7f80fd0e8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80f81f0df0 con 0x7f80f806f570 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.849+0000 7f80fd0e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f80f81e7490 con 0x7f80f815e040 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.849+0000 7f80fd0e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f80f81f0170 con 0x7f80f815dc60 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.850+0000 7f80f659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 0x7f80f8153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:13.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.850+0000 7f80f6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80f806f570 0x7f80f81e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:13.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.850+0000 7f80f759e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80f815e040 0x7f80f80c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:13.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.850+0000 7f80f659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 0x7f80f8153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51310/0 (socket says 172.21.15.129:51310) 2024-03-23T15:51:13.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.850+0000 7f80f659c640 1 -- 172.21.15.129:0/1522976751 learned_addr learned my addr 172.21.15.129:0/1522976751 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:13.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.853+0000 7f80f659c640 1 -- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80f815e040 msgr2=0x7f80f80c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:13.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.853+0000 7f80f659c640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80f815e040 0x7f80f80c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:13.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.853+0000 7f80f659c640 1 -- 172.21.15.129:0/1522976751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80f806f570 msgr2=0x7f80f81e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:13.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.853+0000 7f80f659c640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80f806f570 0x7f80f81e6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:13.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.853+0000 7f80f659c640 1 -- 172.21.15.129:0/1522976751 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec0094d0 con 0x7f80f815dc60 2024-03-23T15:51:13.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.856+0000 7f80f659c640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 0x7f80f8153870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0090b0 tx=0x7f80ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:13.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.860+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec03b940 con 0x7f80f815dc60 2024-03-23T15:51:13.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.860+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ec03bac0 con 0x7f80f815dc60 2024-03-23T15:51:13.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.860+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec03be70 con 0x7f80f815dc60 2024-03-23T15:51:13.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.860+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f81f0450 con 0x7f80f815dc60 2024-03-23T15:51:13.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.860+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f81f0a10 con 0x7f80f815dc60 2024-03-23T15:51:13.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:13.862+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f81f0ba0 con 0x7f80f815dc60 2024-03-23T15:51:14.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.150+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f80ec02f050 con 0x7f80f815dc60 2024-03-23T15:51:14.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.151+0000 7f80df7fe640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80e406b860 0x7f80e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:14.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.151+0000 7f80f6d9d640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80e406b860 0x7f80e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:14.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.151+0000 7f80f6d9d640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80e406b860 0x7f80e406dd20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009aa0 tx=0x7f80e8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:14.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.154+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f80ec07ab30 con 0x7f80f815dc60 2024-03-23T15:51:14.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.165+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f80ec076dd0 con 0x7f80f815dc60 2024-03-23T15:51:14.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.387+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80f80cdca0 con 0x7f80f815dc60 2024-03-23T15:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.402+0000 7f80df7fe640 1 -- 172.21.15.129:0/1522976751 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f80ec07a490 con 0x7f80f815dc60 2024-03-23T15:51:14.405 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:14.405 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80e406b860 msgr2=0x7f80e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80e406b860 0x7f80e406dd20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009aa0 tx=0x7f80e8009190 comp rx=0 tx=0).stop 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 msgr2=0x7f80f8153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 0x7f80f8153870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0090b0 tx=0x7f80ec0046c0 comp rx=0 tx=0).stop 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 shutdown_connections 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80e406b860 0x7f80e406dd20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80f815e040 0x7f80f80c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80f815dc60 0x7f80f8153870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 --2- 172.21.15.129:0/1522976751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80f806f570 0x7f80f81e6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 >> 172.21.15.129:0/1522976751 conn(0x7f80f8071300 msgr2=0x7f80f80c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 shutdown_connections 2024-03-23T15:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:14.404+0000 7f80fd0e8640 1 -- 172.21.15.129:0/1522976751 wait complete. 2024-03-23T15:51:15.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:15.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:16.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:16.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.122+0000 7f08e442c640 1 Processor -- start 2024-03-23T15:51:16.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.123+0000 7f08e442c640 1 -- start start 2024-03-23T15:51:16.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.123+0000 7f08e442c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f08dc0c7ca0 0x7f08dc06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:16.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.123+0000 7f08e442c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08dc15e270 0x7f08dc0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:16.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.124+0000 7f08e442c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08dc15aa90 con 0x7f08dc1544d0 2024-03-23T15:51:16.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.124+0000 7f08e442c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f08dc15de50 con 0x7f08dc0c7ca0 2024-03-23T15:51:16.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.124+0000 7f08e442c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f08dc0c4a70 con 0x7f08dc15e270 2024-03-23T15:51:16.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.124+0000 7f08e19c7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f08dc0c7ca0 0x7f08dc06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:16.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.124+0000 7f08e29c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08dc15e270 0x7f08dc0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:16.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.125+0000 7f08e19c7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f08dc0c7ca0 0x7f08dc06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56970/0 (socket says 172.21.15.129:56970) 2024-03-23T15:51:16.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.125+0000 7f08e19c7640 1 -- 172.21.15.129:0/2502354116 learned_addr learned my addr 172.21.15.129:0/2502354116 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:16.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.127+0000 7f08e19c7640 1 -- 172.21.15.129:0/2502354116 >> v2:172.21.15.157:3300/0 conn(0x7f08dc15e270 msgr2=0x7f08dc0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:16.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.127+0000 7f08e19c7640 1 --2- 172.21.15.129:0/2502354116 >> v2:172.21.15.157:3300/0 conn(0x7f08dc15e270 0x7f08dc0ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.127+0000 7f08e19c7640 1 -- 172.21.15.129:0/2502354116 >> v1:172.21.15.157:6789/0 conn(0x7f08dc1544d0 legacy=0x7f08dc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:16.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.127+0000 7f08e19c7640 1 -- 172.21.15.129:0/2502354116 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc15ad70 con 0x7f08dc0c7ca0 2024-03-23T15:51:16.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.129+0000 7f08e19c7640 1 --2- 172.21.15.129:0/2502354116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc0c7ca0 0x7f08dc06bb70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f08dc06bb00 tx=0x7f08cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6f6c83babf281 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:16.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.133+0000 7f08e11c6640 1 -- 172.21.15.129:0/2502354116 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc031870 con 0x7f08dc0c7ca0 2024-03-23T15:51:16.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.133+0000 7f08e11c6640 1 -- 172.21.15.129:0/2502354116 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08cc0319d0 con 0x7f08dc0c7ca0 2024-03-23T15:51:16.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.133+0000 7f08e11c6640 1 -- 172.21.15.129:0/2502354116 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc031d60 con 0x7f08dc0c7ca0 2024-03-23T15:51:16.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 -- 172.21.15.129:0/2502354116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc0c7ca0 msgr2=0x7f08dc06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:16.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 --2- 172.21.15.129:0/2502354116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc0c7ca0 0x7f08dc06bb70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f08dc06bb00 tx=0x7f08cc02b910 comp rx=0 tx=0).stop 2024-03-23T15:51:16.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 -- 172.21.15.129:0/2502354116 shutdown_connections 2024-03-23T15:51:16.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 --2- 172.21.15.129:0/2502354116 >> v2:172.21.15.157:3300/0 conn(0x7f08dc15e270 0x7f08dc0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 --2- 172.21.15.129:0/2502354116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc0c7ca0 0x7f08dc06bb70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 -- 172.21.15.129:0/2502354116 >> 172.21.15.129:0/2502354116 conn(0x7f08dc06e230 msgr2=0x7f08dc06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:16.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 -- 172.21.15.129:0/2502354116 shutdown_connections 2024-03-23T15:51:16.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.134+0000 7f08e442c640 1 -- 172.21.15.129:0/2502354116 wait complete. 2024-03-23T15:51:16.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.136+0000 7f08e442c640 1 Processor -- start 2024-03-23T15:51:16.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.136+0000 7f08e442c640 1 -- start start 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.137+0000 7f08e442c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08dc0c7ca0 0x7f08dc1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.137+0000 7f08e442c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08dc1544d0 0x7f08dc15cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.137+0000 7f08e442c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 0x7f08dc1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.137+0000 7f08e442c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc1531e0 con 0x7f08dc15e270 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.137+0000 7f08e442c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc1ef9a0 con 0x7f08dc1544d0 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.137+0000 7f08e442c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08dc1554c0 con 0x7f08dc0c7ca0 2024-03-23T15:51:16.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.138+0000 7f08e29c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 0x7f08dc1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:16.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.138+0000 7f08e19c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08dc1544d0 0x7f08dc15cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:16.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.138+0000 7f08e29c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 0x7f08dc1efe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56982/0 (socket says 172.21.15.129:56982) 2024-03-23T15:51:16.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.138+0000 7f08e29c9640 1 -- 172.21.15.129:0/3864249909 learned_addr learned my addr 172.21.15.129:0/3864249909 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:16.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.139+0000 7f08e21c8640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08dc0c7ca0 0x7f08dc1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:16.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.142+0000 7f08e29c9640 1 -- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08dc0c7ca0 msgr2=0x7f08dc1575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:16.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.142+0000 7f08e29c9640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08dc0c7ca0 0x7f08dc1575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.142+0000 7f08e29c9640 1 -- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08dc1544d0 msgr2=0x7f08dc15cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:16.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.142+0000 7f08e29c9640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08dc1544d0 0x7f08dc15cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.142+0000 7f08e29c9640 1 -- 172.21.15.129:0/3864249909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08cc0094d0 con 0x7f08dc15e270 2024-03-23T15:51:16.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.146+0000 7f08e29c9640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 0x7f08dc1efe30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009d30 tx=0x7f08d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.150+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4009ba0 con 0x7f08dc15e270 2024-03-23T15:51:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.150+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08d400f040 con 0x7f08dc15e270 2024-03-23T15:51:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.150+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08dc153430 con 0x7f08dc15e270 2024-03-23T15:51:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.150+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d40146c0 con 0x7f08dc15e270 2024-03-23T15:51:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.150+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08dc1551e0 con 0x7f08dc15e270 2024-03-23T15:51:16.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.151+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08dc06b6a0 con 0x7f08dc15e270 2024-03-23T15:51:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.405+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f08d4014820 con 0x7f08dc15e270 2024-03-23T15:51:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.405+0000 7f08d2ffd640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f08c006b860 0x7f08c006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.406+0000 7f08e21c8640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f08c006b860 0x7f08c006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.406+0000 7f08e21c8640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f08c006b860 0x7f08c006dd20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f08d8004f60 tx=0x7f08d8007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.421+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f08d4059ba0 con 0x7f08dc15e270 2024-03-23T15:51:16.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.421+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08d4055e40 con 0x7f08dc15e270 2024-03-23T15:51:16.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.652+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08dc075740 con 0x7f08dc15e270 2024-03-23T15:51:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.704+0000 7f08d2ffd640 1 -- 172.21.15.129:0/3864249909 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f08d4018020 con 0x7f08dc15e270 2024-03-23T15:51:16.707 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:16.707 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f08c006b860 msgr2=0x7f08c006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f08c006b860 0x7f08c006dd20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f08d8004f60 tx=0x7f08d8007e30 comp rx=0 tx=0).stop 2024-03-23T15:51:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 msgr2=0x7f08dc1efe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 0x7f08dc1efe30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009d30 tx=0x7f08d400c490 comp rx=0 tx=0).stop 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 shutdown_connections 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08dc15e270 0x7f08dc1efe30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f08c006b860 0x7f08c006dd20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08dc1544d0 0x7f08dc15cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 --2- 172.21.15.129:0/3864249909 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08dc0c7ca0 0x7f08dc1575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 >> 172.21.15.129:0/3864249909 conn(0x7f08dc06e230 msgr2=0x7f08dc0c80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 shutdown_connections 2024-03-23T15:51:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:16.706+0000 7f08e442c640 1 -- 172.21.15.129:0/3864249909 wait complete. 2024-03-23T15:51:17.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:17.328 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:18.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.428+0000 7fb4709d0640 1 Processor -- start 2024-03-23T15:51:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.428+0000 7fb4709d0640 1 -- start start 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.429+0000 7fb4709d0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb46c0c7e70 0x7fb46c1621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.429+0000 7fb4709d0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb46c073560 con 0x7fb46c072ad0 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.429+0000 7fb4709d0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb46c072fc0 con 0x7fb46c150d00 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.429+0000 7fb4709d0640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb46c1510e0 con 0x7fb46c0c7e70 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.429+0000 7fb46ad9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb46c0c7e70 0x7fb46c1621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.430+0000 7fb46a59c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb46c150d00 0x7fb46c1568b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42200/0 (socket says 172.21.15.129:42200) 2024-03-23T15:51:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.430+0000 7fb46a59c640 1 -- 172.21.15.129:0/1773401270 learned_addr learned my addr 172.21.15.129:0/1773401270 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:18.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.433+0000 7fb46959a640 1 -- 172.21.15.129:0/1773401270 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2716821016 0 0) 0x7fb46c072fc0 con 0x7fb46c150d00 2024-03-23T15:51:18.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.433+0000 7fb46959a640 1 -- 172.21.15.129:0/1773401270 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb454003440 con 0x7fb46c150d00 2024-03-23T15:51:18.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.433+0000 7fb46ad9d640 1 -- 172.21.15.129:0/1773401270 >> v1:172.21.15.157:6789/0 conn(0x7fb46c150d00 legacy=0x7fb46c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:18.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.433+0000 7fb46ad9d640 1 -- 172.21.15.129:0/1773401270 >> v1:172.21.15.129:6789/0 conn(0x7fb46c072ad0 legacy=0x7fb46c06d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:18.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.433+0000 7fb46ad9d640 1 -- 172.21.15.129:0/1773401270 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46c073280 con 0x7fb46c0c7e70 2024-03-23T15:51:18.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.439+0000 7fb46ad9d640 1 --2- 172.21.15.129:0/1773401270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c0c7e70 0x7fb46c1621a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb464009d90 tx=0x7fb46402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e65dfd6cb14176 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:18.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.443+0000 7fb46959a640 1 -- 172.21.15.129:0/1773401270 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46402cd00 con 0x7fb46c0c7e70 2024-03-23T15:51:18.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.443+0000 7fb46959a640 1 -- 172.21.15.129:0/1773401270 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46402ce60 con 0x7fb46c0c7e70 2024-03-23T15:51:18.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.443+0000 7fb46959a640 1 -- 172.21.15.129:0/1773401270 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464034660 con 0x7fb46c0c7e70 2024-03-23T15:51:18.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 -- 172.21.15.129:0/1773401270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c0c7e70 msgr2=0x7fb46c1621a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:18.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 --2- 172.21.15.129:0/1773401270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c0c7e70 0x7fb46c1621a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb464009d90 tx=0x7fb46402bf50 comp rx=0 tx=0).stop 2024-03-23T15:51:18.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 -- 172.21.15.129:0/1773401270 shutdown_connections 2024-03-23T15:51:18.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 --2- 172.21.15.129:0/1773401270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c0c7e70 0x7fb46c1621a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:18.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 -- 172.21.15.129:0/1773401270 >> 172.21.15.129:0/1773401270 conn(0x7fb46c0c8370 msgr2=0x7fb46c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:18.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 -- 172.21.15.129:0/1773401270 shutdown_connections 2024-03-23T15:51:18.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.444+0000 7fb4709d0640 1 -- 172.21.15.129:0/1773401270 wait complete. 2024-03-23T15:51:18.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.446+0000 7fb4709d0640 1 Processor -- start 2024-03-23T15:51:18.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.446+0000 7fb4709d0640 1 -- start start 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.447+0000 7fb4709d0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 0x7fb46c14e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.447+0000 7fb4709d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb46c0c7e70 0x7fb46c14bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.447+0000 7fb4709d0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb46c150d00 0x7fb46c0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.447+0000 7fb4709d0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c14c1e0 con 0x7fb46c072ad0 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.447+0000 7fb4709d0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c146ba0 con 0x7fb46c150d00 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.447+0000 7fb4709d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb46c1e0ea0 con 0x7fb46c0c7e70 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.448+0000 7fb46a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 0x7fb46c14e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.448+0000 7fb469d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb46c0c7e70 0x7fb46c14bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.448+0000 7fb46ad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb46c150d00 0x7fb46c0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.448+0000 7fb46a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 0x7fb46c14e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56988/0 (socket says 172.21.15.129:56988) 2024-03-23T15:51:18.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.448+0000 7fb46a59c640 1 -- 172.21.15.129:0/3147779770 learned_addr learned my addr 172.21.15.129:0/3147779770 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:18.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.452+0000 7fb46a59c640 1 -- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb46c0c7e70 msgr2=0x7fb46c14bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.452+0000 7fb46a59c640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb46c0c7e70 0x7fb46c14bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.452+0000 7fb46a59c640 1 -- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb46c150d00 msgr2=0x7fb46c0caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.452+0000 7fb46a59c640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb46c150d00 0x7fb46c0caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.452+0000 7fb46a59c640 1 -- 172.21.15.129:0/3147779770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb464009a70 con 0x7fb46c072ad0 2024-03-23T15:51:18.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.454+0000 7fb46a59c640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 0x7fb46c14e500 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00c390 tx=0x7fb45c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:18.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.458+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c005ce0 con 0x7fb46c072ad0 2024-03-23T15:51:18.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.458+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45c005e40 con 0x7fb46c072ad0 2024-03-23T15:51:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.458+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46c1e1140 con 0x7fb46c072ad0 2024-03-23T15:51:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.458+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0072f0 con 0x7fb46c072ad0 2024-03-23T15:51:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.458+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb46c146730 con 0x7fb46c072ad0 2024-03-23T15:51:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.459+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb46c147050 con 0x7fb46c072ad0 2024-03-23T15:51:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.478+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb45c007450 con 0x7fb46c072ad0 2024-03-23T15:51:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.478+0000 7fb45affd640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb44c06b860 0x7fb44c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.478+0000 7fb469d9b640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb44c06b860 0x7fb44c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:18.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.479+0000 7fb469d9b640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb44c06b860 0x7fb44c06dd20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb46c14ec80 tx=0x7fb460009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:18.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.480+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb45c058230 con 0x7fb46c072ad0 2024-03-23T15:51:18.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.769+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb45c0544d0 con 0x7fb46c072ad0 2024-03-23T15:51:18.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:18.995+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb46c147050 con 0x7fb46c072ad0 2024-03-23T15:51:19.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.031+0000 7fb45affd640 1 -- 172.21.15.129:0/3147779770 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb46c147050 con 0x7fb46c072ad0 2024-03-23T15:51:19.035 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:19.035 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:19.035 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb44c06b860 msgr2=0x7fb44c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb44c06b860 0x7fb44c06dd20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb46c14ec80 tx=0x7fb460009040 comp rx=0 tx=0).stop 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 msgr2=0x7fb46c14e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 0x7fb46c14e500 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00c390 tx=0x7fb45c00c850 comp rx=0 tx=0).stop 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 shutdown_connections 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb44c06b860 0x7fb44c06dd20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:19.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb46c150d00 0x7fb46c0caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb46c0c7e70 0x7fb46c14bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 --2- 172.21.15.129:0/3147779770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb46c072ad0 0x7fb46c14e500 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 >> 172.21.15.129:0/3147779770 conn(0x7fb46c0c8370 msgr2=0x7fb46c074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 shutdown_connections 2024-03-23T15:51:19.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:19.034+0000 7fb4709d0640 1 -- 172.21.15.129:0/3147779770 wait complete. 2024-03-23T15:51:19.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:19.653 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:20.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.754+0000 7ff688e39640 1 Processor -- start 2024-03-23T15:51:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.754+0000 7ff688e39640 1 -- start start 2024-03-23T15:51:20.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.755+0000 7ff688e39640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff6840c8290 0x7ff684158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:20.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.755+0000 7ff688e39640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff68406f950 0x7ff684159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:20.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.755+0000 7ff688e39640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff68415b5d0 con 0x7ff6841520b0 2024-03-23T15:51:20.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.755+0000 7ff688e39640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff68415c430 con 0x7ff6840c8290 2024-03-23T15:51:20.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.755+0000 7ff688e39640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff68406fd60 con 0x7ff68406f950 2024-03-23T15:51:20.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.756+0000 7ff68259c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff6841520b0 0x7ff6840cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:33250/0 (socket says 172.21.15.129:33250) 2024-03-23T15:51:20.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.756+0000 7ff68259c640 1 -- 172.21.15.129:0/4149979579 learned_addr learned my addr 172.21.15.129:0/4149979579 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:20.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.756+0000 7ff682d9d640 1 --2- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff68406f950 0x7ff684159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:20.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.756+0000 7ff681d9b640 1 --2- 172.21.15.129:0/4149979579 >> v2:172.21.15.157:3300/0 conn(0x7ff6840c8290 0x7ff684158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.758+0000 7ff68159a640 1 -- 172.21.15.129:0/4149979579 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574015193 0 0) 0x7ff68415b5d0 con 0x7ff6841520b0 2024-03-23T15:51:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.758+0000 7ff68159a640 1 -- 172.21.15.129:0/4149979579 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff66c003520 con 0x7ff6841520b0 2024-03-23T15:51:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.758+0000 7ff681d9b640 1 -- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff68406f950 msgr2=0x7ff684159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.758+0000 7ff681d9b640 1 --2- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff68406f950 0x7ff684159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.758+0000 7ff681d9b640 1 -- 172.21.15.129:0/4149979579 >> v1:172.21.15.129:6789/0 conn(0x7ff6841520b0 legacy=0x7ff6840cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.758+0000 7ff681d9b640 1 -- 172.21.15.129:0/4149979579 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff684152490 con 0x7ff6840c8290 2024-03-23T15:51:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.760+0000 7ff681d9b640 1 --2- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6840c8290 0x7ff684158a60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff6780097a0 tx=0x7ff67802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be3a6e7c637075ce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:20.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.763+0000 7ff68159a640 1 -- 172.21.15.129:0/4149979579 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678030610 con 0x7ff6840c8290 2024-03-23T15:51:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff68159a640 1 -- 172.21.15.129:0/4149979579 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff678030770 con 0x7ff6840c8290 2024-03-23T15:51:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff68159a640 1 -- 172.21.15.129:0/4149979579 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678030ae0 con 0x7ff6840c8290 2024-03-23T15:51:20.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff688e39640 1 -- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6840c8290 msgr2=0x7ff684158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff688e39640 1 --2- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6840c8290 0x7ff684158a60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff6780097a0 tx=0x7ff67802b960 comp rx=0 tx=0).stop 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff688e39640 1 -- 172.21.15.129:0/4149979579 shutdown_connections 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff688e39640 1 --2- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff68406f950 0x7ff684159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff688e39640 1 --2- 172.21.15.129:0/4149979579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6840c8290 0x7ff684158a60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.765+0000 7ff688e39640 1 -- 172.21.15.129:0/4149979579 >> 172.21.15.129:0/4149979579 conn(0x7ff684061b50 msgr2=0x7ff684061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.766+0000 7ff688e39640 1 -- 172.21.15.129:0/4149979579 shutdown_connections 2024-03-23T15:51:20.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.766+0000 7ff688e39640 1 -- 172.21.15.129:0/4149979579 wait complete. 2024-03-23T15:51:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.767+0000 7ff688e39640 1 Processor -- start 2024-03-23T15:51:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.767+0000 7ff688e39640 1 -- start start 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff688e39640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 0x7ff6840c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff688e39640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6840c8290 0x7ff6840c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff688e39640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6841520b0 0x7ff6841f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff688e39640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff68406e270 con 0x7ff68406f950 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff688e39640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6840c6c30 con 0x7ff6841520b0 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff688e39640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff6840c5b40 con 0x7ff6840c8290 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff68259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 0x7ff6840c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff682d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6841520b0 0x7ff6841f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:20.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.768+0000 7ff681d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6840c8290 0x7ff6840c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:20.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.769+0000 7ff68259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 0x7ff6840c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53440/0 (socket says 172.21.15.129:53440) 2024-03-23T15:51:20.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.769+0000 7ff68259c640 1 -- 172.21.15.129:0/3920960444 learned_addr learned my addr 172.21.15.129:0/3920960444 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.772+0000 7ff68259c640 1 -- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6840c8290 msgr2=0x7ff6840c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.772+0000 7ff68259c640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6840c8290 0x7ff6840c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.772+0000 7ff68259c640 1 -- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6841520b0 msgr2=0x7ff6841f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.772+0000 7ff68259c640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6841520b0 0x7ff6841f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.772+0000 7ff68259c640 1 -- 172.21.15.129:0/3920960444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff678009400 con 0x7ff68406f950 2024-03-23T15:51:20.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.774+0000 7ff68259c640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 0x7ff6840c9a20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff67c00cad0 tx=0x7ff67c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:20.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.778+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c00ac30 con 0x7ff68406f950 2024-03-23T15:51:20.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.778+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67c00ad90 con 0x7ff68406f950 2024-03-23T15:51:20.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.778+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68406d6f0 con 0x7ff68406f950 2024-03-23T15:51:20.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.778+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c012620 con 0x7ff68406f950 2024-03-23T15:51:20.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.778+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68406dcb0 con 0x7ff68406f950 2024-03-23T15:51:20.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:20.779+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6841f3480 con 0x7ff68406f950 2024-03-23T15:51:21.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.034+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff67c007070 con 0x7ff68406f950 2024-03-23T15:51:21.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.034+0000 7ff672ffd640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff66406b810 0x7ff66406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:21.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.035+0000 7ff681d9b640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff66406b810 0x7ff66406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:21.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.035+0000 7ff681d9b640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff66406b810 0x7ff66406dcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff678036000 tx=0x7ff678008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:21.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.049+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff67c058bb0 con 0x7ff68406f950 2024-03-23T15:51:21.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.049+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff67c054e50 con 0x7ff68406f950 2024-03-23T15:51:21.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.279+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6841f3480 con 0x7ff68406f950 2024-03-23T15:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.308+0000 7ff672ffd640 1 -- 172.21.15.129:0/3920960444 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff6841f3480 con 0x7ff68406f950 2024-03-23T15:51:21.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:21.312 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:21.312 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:21.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff66406b810 msgr2=0x7ff66406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:21.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff66406b810 0x7ff66406dcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff678036000 tx=0x7ff678008030 comp rx=0 tx=0).stop 2024-03-23T15:51:21.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 msgr2=0x7ff6840c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:21.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 0x7ff6840c9a20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff67c00cad0 tx=0x7ff67c00cf90 comp rx=0 tx=0).stop 2024-03-23T15:51:21.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 shutdown_connections 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff66406b810 0x7ff66406dcd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6841520b0 0x7ff6841f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6840c8290 0x7ff6840c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 --2- 172.21.15.129:0/3920960444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff68406f950 0x7ff6840c9a20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 >> 172.21.15.129:0/3920960444 conn(0x7ff684061b50 msgr2=0x7ff6841512d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 shutdown_connections 2024-03-23T15:51:21.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:21.311+0000 7ff688e39640 1 -- 172.21.15.129:0/3920960444 wait complete. 2024-03-23T15:51:21.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:21.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:22.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:23.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.030+0000 7f5d8fbc7640 1 Processor -- start 2024-03-23T15:51:23.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.030+0000 7f5d8fbc7640 1 -- start start 2024-03-23T15:51:23.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.031+0000 7f5d8fbc7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5d880cd820 0x7f5d88158c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:23.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.031+0000 7f5d8fbc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cdc00 0x7f5d8806fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:23.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.031+0000 7f5d8fbc7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d880cba60 con 0x7f5d880ce020 2024-03-23T15:51:23.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.031+0000 7f5d8fbc7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5d88070e10 con 0x7f5d880cd820 2024-03-23T15:51:23.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.031+0000 7f5d8fbc7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5d881591a0 con 0x7f5d880cdc00 2024-03-23T15:51:23.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.032+0000 7f5d8d162640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cdc00 0x7f5d8806fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:23.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.032+0000 7f5d8d963640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5d880cd820 0x7f5d88158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:23.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.033+0000 7f5d8d162640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cdc00 0x7f5d8806fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:43160/0 (socket says 172.21.15.129:43160) 2024-03-23T15:51:23.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.033+0000 7f5d8d162640 1 -- 172.21.15.129:0/2840901379 learned_addr learned my addr 172.21.15.129:0/2840901379 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:23.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.035+0000 7f5d8d963640 1 -- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cdc00 msgr2=0x7f5d8806fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:23.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.035+0000 7f5d8d963640 1 --2- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cdc00 0x7f5d8806fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:23.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.035+0000 7f5d8d963640 1 -- 172.21.15.129:0/2840901379 >> v1:172.21.15.157:6789/0 conn(0x7f5d880ce020 legacy=0x7f5d88151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:23.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.035+0000 7f5d8d963640 1 -- 172.21.15.129:0/2840901379 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d880cd120 con 0x7f5d880cd820 2024-03-23T15:51:23.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.039+0000 7f5d8d963640 1 --2- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cd820 0x7f5d88158c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d840097a0 tx=0x7f5d8402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ffbf545cd597359 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:23.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.043+0000 7f5d8c961640 1 -- 172.21.15.129:0/2840901379 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8402f050 con 0x7f5d880cd820 2024-03-23T15:51:23.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.043+0000 7f5d8c961640 1 -- 172.21.15.129:0/2840901379 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d84031700 con 0x7f5d880cd820 2024-03-23T15:51:23.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.043+0000 7f5d8c961640 1 -- 172.21.15.129:0/2840901379 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d84031a90 con 0x7f5d880cd820 2024-03-23T15:51:23.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cd820 msgr2=0x7f5d88158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:23.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cd820 0x7f5d88158c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d840097a0 tx=0x7f5d8402b7a0 comp rx=0 tx=0).stop 2024-03-23T15:51:23.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2840901379 shutdown_connections 2024-03-23T15:51:23.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cdc00 0x7f5d8806fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:23.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2840901379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cd820 0x7f5d88158c60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:23.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2840901379 >> 172.21.15.129:0/2840901379 conn(0x7f5d88071300 msgr2=0x7f5d880716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:23.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2840901379 shutdown_connections 2024-03-23T15:51:23.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.044+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2840901379 wait complete. 2024-03-23T15:51:23.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.046+0000 7f5d8fbc7640 1 Processor -- start 2024-03-23T15:51:23.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.046+0000 7f5d8fbc7640 1 -- start start 2024-03-23T15:51:23.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.047+0000 7f5d8fbc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cd820 0x7f5d880c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:23.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.047+0000 7f5d8fbc7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 0x7f5d880c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:23.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.047+0000 7f5d8fbc7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d880ce020 0x7f5d881ed990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:23.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.047+0000 7f5d8fbc7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d880c86e0 con 0x7f5d880ce020 2024-03-23T15:51:23.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.047+0000 7f5d8fbc7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8814e240 con 0x7f5d880cdc00 2024-03-23T15:51:23.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.047+0000 7f5d8fbc7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5d881ee3d0 con 0x7f5d880cd820 2024-03-23T15:51:23.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.048+0000 7f5d8e164640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d880ce020 0x7f5d881ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:23.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.048+0000 7f5d8d162640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 0x7f5d880c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:23.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.048+0000 7f5d8d963640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cd820 0x7f5d880c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:23.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.049+0000 7f5d8d162640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 0x7f5d880c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:33238/0 (socket says 172.21.15.129:33238) 2024-03-23T15:51:23.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.049+0000 7f5d8d162640 1 -- 172.21.15.129:0/2679106072 learned_addr learned my addr 172.21.15.129:0/2679106072 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:23.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.052+0000 7f5d8d162640 1 -- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cd820 msgr2=0x7f5d880c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:23.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.052+0000 7f5d8d162640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cd820 0x7f5d880c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:23.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.052+0000 7f5d8d162640 1 -- 172.21.15.129:0/2679106072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d880ce020 msgr2=0x7f5d881ed990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:23.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.052+0000 7f5d8d162640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d880ce020 0x7f5d881ed990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:23.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.052+0000 7f5d8d162640 1 -- 172.21.15.129:0/2679106072 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d84009400 con 0x7f5d880cdc00 2024-03-23T15:51:23.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.054+0000 7f5d8d162640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 0x7f5d880c81a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e8b0 tx=0x7f5d7800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:23.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.057+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d78009600 con 0x7f5d880cdc00 2024-03-23T15:51:23.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.057+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d881ee6b0 con 0x7f5d880cdc00 2024-03-23T15:51:23.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.057+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d880c79a0 con 0x7f5d880cdc00 2024-03-23T15:51:23.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.058+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d880c7520 con 0x7f5d880cdc00 2024-03-23T15:51:23.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.059+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7800c940 con 0x7f5d880cdc00 2024-03-23T15:51:23.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.059+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800ccb0 con 0x7f5d880cdc00 2024-03-23T15:51:23.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.361+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5d78026480 con 0x7f5d880cdc00 2024-03-23T15:51:23.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.362+0000 7f5d7e7fc640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5d6c06b810 0x7f5d6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:23.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.362+0000 7f5d8d963640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5d6c06b810 0x7f5d6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:23.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.362+0000 7f5d8d963640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5d6c06b810 0x7f5d6c06dcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d84035000 tx=0x7f5d84008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:23.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.368+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5d78059180 con 0x7f5d880cdc00 2024-03-23T15:51:23.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.388+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5d78055420 con 0x7f5d880cdc00 2024-03-23T15:51:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.609+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d880c7520 con 0x7f5d880cdc00 2024-03-23T15:51:23.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.682+0000 7f5d7e7fc640 1 -- 172.21.15.129:0/2679106072 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5d880c7520 con 0x7f5d880cdc00 2024-03-23T15:51:24.178 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:24.178 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.684+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5d6c06b810 msgr2=0x7f5d6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.684+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5d6c06b810 0x7f5d6c06dcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d84035000 tx=0x7f5d84008030 comp rx=0 tx=0).stop 2024-03-23T15:51:24.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.684+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 msgr2=0x7f5d880c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.684+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 0x7f5d880c81a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800e8b0 tx=0x7f5d7800ed70 comp rx=0 tx=0).stop 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 shutdown_connections 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d880ce020 0x7f5d881ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5d6c06b810 0x7f5d6c06dcd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d880cdc00 0x7f5d880c81a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 --2- 172.21.15.129:0/2679106072 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d880cd820 0x7f5d880c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 >> 172.21.15.129:0/2679106072 conn(0x7f5d88071300 msgr2=0x7f5d881577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 shutdown_connections 2024-03-23T15:51:24.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:23.685+0000 7f5d8fbc7640 1 -- 172.21.15.129:0/2679106072 wait complete. 2024-03-23T15:51:24.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:24.786 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:25.788 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:25.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.888+0000 7f777960e640 1 Processor -- start 2024-03-23T15:51:25.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.888+0000 7f777960e640 1 -- start start 2024-03-23T15:51:25.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.889+0000 7f777960e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f77741542e0 0x7f777415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.889+0000 7f777960e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f777406b930 0x7f77740cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.889+0000 7f777960e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406bd70 0x7f77740cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.889+0000 7f777960e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f777415e810 con 0x7f777406b930 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.889+0000 7f777960e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f777415b270 con 0x7f77741542e0 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.889+0000 7f777960e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77741546c0 con 0x7f777406bd70 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.890+0000 7f77727fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f777406b930 0x7f77740cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.890+0000 7f7772ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f77741542e0 0x7f777415ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.890+0000 7f77737fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406bd70 0x7f77740cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.890+0000 7f77727fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f777406b930 0x7f77740cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53456/0 (socket says 172.21.15.129:53456) 2024-03-23T15:51:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.890+0000 7f77727fc640 1 -- 172.21.15.129:0/200166802 learned_addr learned my addr 172.21.15.129:0/200166802 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.892+0000 7f77727fc640 1 -- 172.21.15.129:0/200166802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406bd70 msgr2=0x7f77740cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.892+0000 7f77727fc640 1 --2- 172.21.15.129:0/200166802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406bd70 0x7f77740cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.892+0000 7f77727fc640 1 -- 172.21.15.129:0/200166802 >> v2:172.21.15.157:3300/0 conn(0x7f77741542e0 msgr2=0x7f777415ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.892+0000 7f77727fc640 1 --2- 172.21.15.129:0/200166802 >> v2:172.21.15.157:3300/0 conn(0x7f77741542e0 0x7f777415ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.892+0000 7f77727fc640 1 -- 172.21.15.129:0/200166802 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777415d9b0 con 0x7f777406b930 2024-03-23T15:51:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.894+0000 7f77727fc640 1 --2- 172.21.15.129:0/200166802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406b930 0x7f77740cc780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f77740cd2a0 tx=0x7f776802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74201783fd7df80 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:25.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f7771ffb640 1 -- 172.21.15.129:0/200166802 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768030860 con 0x7f777406b930 2024-03-23T15:51:25.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f7771ffb640 1 -- 172.21.15.129:0/200166802 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77680309c0 con 0x7f777406b930 2024-03-23T15:51:25.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f7771ffb640 1 -- 172.21.15.129:0/200166802 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768030d50 con 0x7f777406b930 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 -- 172.21.15.129:0/200166802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406b930 msgr2=0x7f77740cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 --2- 172.21.15.129:0/200166802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406b930 0x7f77740cc780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f77740cd2a0 tx=0x7f776802b9a0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 -- 172.21.15.129:0/200166802 shutdown_connections 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 --2- 172.21.15.129:0/200166802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406bd70 0x7f77740cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 --2- 172.21.15.129:0/200166802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406b930 0x7f77740cc780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 --2- 172.21.15.129:0/200166802 >> v2:172.21.15.157:3300/0 conn(0x7f77741542e0 0x7f777415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 -- 172.21.15.129:0/200166802 >> 172.21.15.129:0/200166802 conn(0x7f777406eb20 msgr2=0x7f777406c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 -- 172.21.15.129:0/200166802 shutdown_connections 2024-03-23T15:51:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.898+0000 7f777960e640 1 -- 172.21.15.129:0/200166802 wait complete. 2024-03-23T15:51:25.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.900+0000 7f777960e640 1 Processor -- start 2024-03-23T15:51:25.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.900+0000 7f777960e640 1 -- start start 2024-03-23T15:51:25.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f777960e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406b930 0x7f77741f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:25.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f777960e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 0x7f77741586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:25.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f777960e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77741542e0 0x7f777415b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:25.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f777960e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77741f0740 con 0x7f777406bd70 2024-03-23T15:51:25.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f777960e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f777415bc20 con 0x7f77741542e0 2024-03-23T15:51:25.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f777960e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f777415bf00 con 0x7f777406b930 2024-03-23T15:51:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f77727fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 0x7f77741586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f7772ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406b930 0x7f77741f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f77737fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77741542e0 0x7f777415b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f77727fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 0x7f77741586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53466/0 (socket says 172.21.15.129:53466) 2024-03-23T15:51:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.901+0000 7f77727fc640 1 -- 172.21.15.129:0/2496136427 learned_addr learned my addr 172.21.15.129:0/2496136427 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.905+0000 7f77727fc640 1 -- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406b930 msgr2=0x7f77741f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.905+0000 7f77727fc640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406b930 0x7f77741f0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.905+0000 7f77727fc640 1 -- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77741542e0 msgr2=0x7f777415b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.905+0000 7f77727fc640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77741542e0 0x7f777415b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.905+0000 7f77727fc640 1 -- 172.21.15.129:0/2496136427 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77680094d0 con 0x7f777406bd70 2024-03-23T15:51:25.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.907+0000 7f77727fc640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 0x7f77741586f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f776802b9e0 tx=0x7f7768004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:25.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.910+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768039a90 con 0x7f777406bd70 2024-03-23T15:51:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.910+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7768039c10 con 0x7f777406bd70 2024-03-23T15:51:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.910+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777415c1e0 con 0x7f777406bd70 2024-03-23T15:51:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.910+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768043410 con 0x7f777406bd70 2024-03-23T15:51:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.910+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777415b940 con 0x7f777406bd70 2024-03-23T15:51:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:25.912+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77741f08f0 con 0x7f777406bd70 2024-03-23T15:51:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.165+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f776804c3b0 con 0x7f777406bd70 2024-03-23T15:51:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.166+0000 7f775b7fe640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f776006b860 0x7f776006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.166+0000 7f7772ffd640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f776006b860 0x7f776006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:26.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.167+0000 7f7772ffd640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f776006b860 0x7f776006dd20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7764009b40 tx=0x7f7764009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:26.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.179+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f776807b470 con 0x7f777406bd70 2024-03-23T15:51:26.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.180+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7768077710 con 0x7f777406bd70 2024-03-23T15:51:26.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.398+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77741f08f0 con 0x7f777406bd70 2024-03-23T15:51:26.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.421+0000 7f775b7fe640 1 -- 172.21.15.129:0/2496136427 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f77741f08f0 con 0x7f777406bd70 2024-03-23T15:51:26.425 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:26.425 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:26.425 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f776006b860 msgr2=0x7f776006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f776006b860 0x7f776006dd20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7764009b40 tx=0x7f7764009190 comp rx=0 tx=0).stop 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 msgr2=0x7f77741586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 0x7f77741586f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f776802b9e0 tx=0x7f7768004730 comp rx=0 tx=0).stop 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 shutdown_connections 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f776006b860 0x7f776006dd20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77741542e0 0x7f777415b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:26.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f777406bd70 0x7f77741586f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:26.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 --2- 172.21.15.129:0/2496136427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f777406b930 0x7f77741f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:26.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 >> 172.21.15.129:0/2496136427 conn(0x7f777406eb20 msgr2=0x7f7774151b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:26.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 shutdown_connections 2024-03-23T15:51:26.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:26.424+0000 7f777960e640 1 -- 172.21.15.129:0/2496136427 wait complete. 2024-03-23T15:51:27.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:27.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:28.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:28.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.144+0000 7fc670055640 1 Processor -- start 2024-03-23T15:51:28.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.144+0000 7fc670055640 1 -- start start 2024-03-23T15:51:28.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.145+0000 7fc670055640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc6680ca4a0 0x7fc66806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.145+0000 7fc670055640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6680cb810 con 0x7fc66815dda0 2024-03-23T15:51:28.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.145+0000 7fc670055640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6680cc3b0 con 0x7fc66815d9c0 2024-03-23T15:51:28.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.145+0000 7fc670055640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc6680cac00 con 0x7fc6680ca4a0 2024-03-23T15:51:28.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.145+0000 7fc66e5f2640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc66815dda0 0x7fc66815e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:33264/0 (socket says 172.21.15.129:33264) 2024-03-23T15:51:28.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.145+0000 7fc66e5f2640 1 -- 172.21.15.129:0/128891930 learned_addr learned my addr 172.21.15.129:0/128891930 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:28.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.146+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/128891930 >> v2:172.21.15.157:3300/0 conn(0x7fc6680ca4a0 0x7fc66806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.148+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599385037 0 0) 0x7fc6680cb810 con 0x7fc66815dda0 2024-03-23T15:51:28.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.148+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc654003520 con 0x7fc66815dda0 2024-03-23T15:51:28.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.150+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589533663 0 0) 0x7fc6680cc3b0 con 0x7fc66815d9c0 2024-03-23T15:51:28.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.150+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6680cb810 con 0x7fc66815d9c0 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3595687091 0 0) 0x7fc654003520 con 0x7fc66815dda0 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 >> v2:172.21.15.157:3300/0 conn(0x7fc6680ca4a0 msgr2=0x7fc66806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 --2- 172.21.15.129:0/128891930 >> v2:172.21.15.157:3300/0 conn(0x7fc6680ca4a0 0x7fc66806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 >> v1:172.21.15.157:6789/0 conn(0x7fc66815d9c0 legacy=0x7fc6680cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6680caee0 con 0x7fc66815dda0 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7fc6600019f0 con 0x7fc66815dda0 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 >> v1:172.21.15.129:6789/0 conn(0x7fc66815dda0 legacy=0x7fc66815e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc654003b50 0x7fc654003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6540047c0 0x7fc654004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc654005180 0x7fc65400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6680cac00 con 0x7fc654005180 2024-03-23T15:51:28.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc65400c050 con 0x7fc654003b50 2024-03-23T15:51:28.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.151+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc65400c210 con 0x7fc6540047c0 2024-03-23T15:51:28.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.152+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6540047c0 0x7fc654004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.154+0000 7fc66e5f2640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc654005180 0x7fc65400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.155+0000 7fc66d5f0640 1 -- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc654003b50 msgr2=0x7fc654003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:28.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.155+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc654003b50 0x7fc654003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.155+0000 7fc66d5f0640 1 -- 172.21.15.129:0/128891930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc654005180 msgr2=0x7fc65400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:28.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.155+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc654005180 0x7fc65400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.155+0000 7fc66e5f2640 1 -- 172.21.15.129:0/128891930 reap_dead start 2024-03-23T15:51:28.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.155+0000 7fc66d5f0640 1 -- 172.21.15.129:0/128891930 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc654003520 con 0x7fc6540047c0 2024-03-23T15:51:28.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.157+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6540047c0 0x7fc654004c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc658009ac0 tx=0x7fc6540057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd272bd658c4d26e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:28.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.160+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65802c6b0 con 0x7fc6540047c0 2024-03-23T15:51:28.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.162+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65802c810 con 0x7fc6540047c0 2024-03-23T15:51:28.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.162+0000 7fc66cdef640 1 -- 172.21.15.129:0/128891930 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65802cba0 con 0x7fc6540047c0 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 -- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6540047c0 msgr2=0x7fc654004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6540047c0 0x7fc654004c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc658009ac0 tx=0x7fc6540057f0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 -- 172.21.15.129:0/128891930 shutdown_connections 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 --2- 172.21.15.129:0/128891930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6540047c0 0x7fc654004c40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 -- 172.21.15.129:0/128891930 >> 172.21.15.129:0/128891930 conn(0x7fc6680cfb20 msgr2=0x7fc6680cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 -- 172.21.15.129:0/128891930 shutdown_connections 2024-03-23T15:51:28.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.163+0000 7fc670055640 1 -- 172.21.15.129:0/128891930 wait complete. 2024-03-23T15:51:28.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.164+0000 7fc670055640 1 Processor -- start 2024-03-23T15:51:28.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.164+0000 7fc670055640 1 -- start start 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc670055640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 0x7fc668159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc670055640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6680b2740 0x7fc668156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc670055640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6680b3cd0 0x7fc6680b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc670055640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6680b38b0 con 0x7fc6540047c0 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc670055640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc6680c3ec0 con 0x7fc6680b2740 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc670055640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc6680b4690 con 0x7fc6680b3cd0 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc66ddf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 0x7fc668159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc66d5f0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6680b2740 0x7fc668156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.165+0000 7fc66ddf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 0x7fc668159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53488/0 (socket says 172.21.15.129:53488) 2024-03-23T15:51:28.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.166+0000 7fc66ddf1640 1 -- 172.21.15.129:0/1540852075 learned_addr learned my addr 172.21.15.129:0/1540852075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:28.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.166+0000 7fc66e5f2640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6680b3cd0 0x7fc6680b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.169+0000 7fc66ddf1640 1 -- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6680b3cd0 msgr2=0x7fc6680b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.169+0000 7fc66ddf1640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6680b3cd0 0x7fc6680b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.169+0000 7fc66ddf1640 1 -- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6680b2740 msgr2=0x7fc668156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.169+0000 7fc66ddf1640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6680b2740 0x7fc668156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.169+0000 7fc66ddf1640 1 -- 172.21.15.129:0/1540852075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66815a780 con 0x7fc6540047c0 2024-03-23T15:51:28.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.171+0000 7fc66ddf1640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 0x7fc668159ca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc66400c760 tx=0x7fc66400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:28.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.175+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640052d0 con 0x7fc6540047c0 2024-03-23T15:51:28.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.175+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc664005430 con 0x7fc6540047c0 2024-03-23T15:51:28.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.175+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664003830 con 0x7fc6540047c0 2024-03-23T15:51:28.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.175+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66815aa40 con 0x7fc6540047c0 2024-03-23T15:51:28.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.175+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66815b000 con 0x7fc6540047c0 2024-03-23T15:51:28.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.176+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc668075740 con 0x7fc6540047c0 2024-03-23T15:51:28.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.433+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc664003990 con 0x7fc6540047c0 2024-03-23T15:51:28.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.433+0000 7fc65e7fc640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc64c06b890 0x7fc64c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:28.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.433+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc64c06b890 0x7fc64c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:28.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.434+0000 7fc66d5f0640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc64c06b890 0x7fc64c06dd50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc658036000 tx=0x7fc658030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:28.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.447+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc664059040 con 0x7fc6540047c0 2024-03-23T15:51:28.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.447+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6640552e0 con 0x7fc6540047c0 2024-03-23T15:51:28.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.672+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc66815a1e0 con 0x7fc6540047c0 2024-03-23T15:51:28.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.694+0000 7fc65e7fc640 1 -- 172.21.15.129:0/1540852075 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc6640589a0 con 0x7fc6540047c0 2024-03-23T15:51:28.697 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:28.698 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:28.698 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc64c06b890 msgr2=0x7fc64c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc64c06b890 0x7fc64c06dd50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc658036000 tx=0x7fc658030000 comp rx=0 tx=0).stop 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 msgr2=0x7fc668159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 0x7fc668159ca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc66400c760 tx=0x7fc66400cc20 comp rx=0 tx=0).stop 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 shutdown_connections 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6680b3cd0 0x7fc6680b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc64c06b890 0x7fc64c06dd50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6680b2740 0x7fc668156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 --2- 172.21.15.129:0/1540852075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6540047c0 0x7fc668159ca0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:28.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 >> 172.21.15.129:0/1540852075 conn(0x7fc6680cfb20 msgr2=0x7fc66815ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:28.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 shutdown_connections 2024-03-23T15:51:28.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:28.696+0000 7fc670055640 1 -- 172.21.15.129:0/1540852075 wait complete. 2024-03-23T15:51:29.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:29.315 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:30.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:30.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.421+0000 7f333148f640 1 Processor -- start 2024-03-23T15:51:30.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.421+0000 7f333148f640 1 -- start start 2024-03-23T15:51:30.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.422+0000 7f333148f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f332c1544b0 0x7f332c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.422+0000 7f333148f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f332c0caa00 0x7f332c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.422+0000 7f333148f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c06c0b0 0x7f332c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.422+0000 7f333148f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f332c0c5be0 con 0x7f332c1544b0 2024-03-23T15:51:30.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.422+0000 7f333148f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f332c0cbcb0 con 0x7f332c0caa00 2024-03-23T15:51:30.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.422+0000 7f333148f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f332c15b280 con 0x7f332c06c0b0 2024-03-23T15:51:30.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.423+0000 7f3323fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f332c0caa00 0x7f332c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.423+0000 7f332b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c06c0b0 0x7f332c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.423+0000 7f332affd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f332c1544b0 0x7f332c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.424+0000 7f3323fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f332c0caa00 0x7f332c06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:33326/0 (socket says 172.21.15.129:33326) 2024-03-23T15:51:30.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.424+0000 7f3323fff640 1 -- 172.21.15.129:0/3268531627 learned_addr learned my addr 172.21.15.129:0/3268531627 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:30.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.427+0000 7f3323fff640 1 -- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c06c0b0 msgr2=0x7f332c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:30.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.427+0000 7f3323fff640 1 --2- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c06c0b0 0x7f332c0c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.427+0000 7f3323fff640 1 -- 172.21.15.129:0/3268531627 >> v2:172.21.15.129:3300/0 conn(0x7f332c1544b0 msgr2=0x7f332c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:30.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.427+0000 7f3323fff640 1 --2- 172.21.15.129:0/3268531627 >> v2:172.21.15.129:3300/0 conn(0x7f332c1544b0 0x7f332c15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.427+0000 7f3323fff640 1 -- 172.21.15.129:0/3268531627 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c15a9e0 con 0x7f332c0caa00 2024-03-23T15:51:30.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.430+0000 7f3323fff640 1 --2- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c06bb70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f332c06ce60 tx=0x7f331402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40dd7855149990b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:30.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.434+0000 7f332a7fc640 1 -- 172.21.15.129:0/3268531627 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314030860 con 0x7f332c0caa00 2024-03-23T15:51:30.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.434+0000 7f332a7fc640 1 -- 172.21.15.129:0/3268531627 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33140309c0 con 0x7f332c0caa00 2024-03-23T15:51:30.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.434+0000 7f332a7fc640 1 -- 172.21.15.129:0/3268531627 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314030d50 con 0x7f332c0caa00 2024-03-23T15:51:30.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 -- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 msgr2=0x7f332c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:30.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 --2- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c06bb70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f332c06ce60 tx=0x7f331402b9a0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 -- 172.21.15.129:0/3268531627 shutdown_connections 2024-03-23T15:51:30.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 --2- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c06c0b0 0x7f332c0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 --2- 172.21.15.129:0/3268531627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c06bb70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 --2- 172.21.15.129:0/3268531627 >> v2:172.21.15.129:3300/0 conn(0x7f332c1544b0 0x7f332c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 -- 172.21.15.129:0/3268531627 >> 172.21.15.129:0/3268531627 conn(0x7f332c061b50 msgr2=0x7f332c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:30.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 -- 172.21.15.129:0/3268531627 shutdown_connections 2024-03-23T15:51:30.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.435+0000 7f333148f640 1 -- 172.21.15.129:0/3268531627 wait complete. 2024-03-23T15:51:30.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.437+0000 7f333148f640 1 Processor -- start 2024-03-23T15:51:30.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.437+0000 7f333148f640 1 -- start start 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.438+0000 7f333148f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c06c0b0 0x7f332c0ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.438+0000 7f333148f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.438+0000 7f333148f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c1544b0 0x7f332c0d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.438+0000 7f333148f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f332c143260 con 0x7f332c06c0b0 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f333148f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f332c0ce6d0 con 0x7f332c0caa00 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f333148f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f332c0ce9b0 con 0x7f332c1544b0 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f332affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c06c0b0 0x7f332c0ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f3323fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f332b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c1544b0 0x7f332c0d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f3323fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c0cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:33332/0 (socket says 172.21.15.129:33332) 2024-03-23T15:51:30.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.439+0000 7f3323fff640 1 -- 172.21.15.129:0/73671609 learned_addr learned my addr 172.21.15.129:0/73671609 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:30.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.444+0000 7f332affd640 1 -- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c1544b0 msgr2=0x7f332c0d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:30.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.444+0000 7f332affd640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c1544b0 0x7f332c0d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.444+0000 7f332affd640 1 -- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 msgr2=0x7f332c0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:30.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.444+0000 7f332affd640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c0cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:30.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.444+0000 7f332affd640 1 -- 172.21.15.129:0/73671609 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33140094d0 con 0x7f332c06c0b0 2024-03-23T15:51:30.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.447+0000 7f332affd640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c06c0b0 0x7f332c0ce0b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f331c00e870 tx=0x7f331c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:30.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.451+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331c00ee60 con 0x7f332c06c0b0 2024-03-23T15:51:30.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.451+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f331c00c940 con 0x7f332c06c0b0 2024-03-23T15:51:30.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.451+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c0cec90 con 0x7f332c06c0b0 2024-03-23T15:51:30.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.451+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331c00ccd0 con 0x7f332c06c0b0 2024-03-23T15:51:30.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.451+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c0cf2d0 con 0x7f332c06c0b0 2024-03-23T15:51:30.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.452+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332c06c7d0 con 0x7f332c06c0b0 2024-03-23T15:51:30.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.474+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f331c013470 con 0x7f332c06c0b0 2024-03-23T15:51:30.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.475+0000 7f33237fe640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f330c06b860 0x7f330c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:30.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.475+0000 7f3323fff640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f330c06b860 0x7f330c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:30.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.476+0000 7f3323fff640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f330c06b860 0x7f330c06dd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f332c0b24c0 tx=0x7f3314008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:30.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.774+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f331c014e40 con 0x7f332c06c0b0 2024-03-23T15:51:30.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:30.775+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f331c055480 con 0x7f332c06c0b0 2024-03-23T15:51:31.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.000+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f332c075740 con 0x7f332c06c0b0 2024-03-23T15:51:31.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.020+0000 7f33237fe640 1 -- 172.21.15.129:0/73671609 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f331c021020 con 0x7f332c06c0b0 2024-03-23T15:51:31.024 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:31.024 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:31.024 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f330c06b860 msgr2=0x7f330c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f330c06b860 0x7f330c06dd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f332c0b24c0 tx=0x7f3314008030 comp rx=0 tx=0).stop 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c06c0b0 msgr2=0x7f332c0ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c06c0b0 0x7f332c0ce0b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f331c00e870 tx=0x7f331c00ed30 comp rx=0 tx=0).stop 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 shutdown_connections 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f332c1544b0 0x7f332c0d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f330c06b860 0x7f330c06dd20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f332c0caa00 0x7f332c0cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 --2- 172.21.15.129:0/73671609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c06c0b0 0x7f332c0ce0b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:31.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 >> 172.21.15.129:0/73671609 conn(0x7f332c061b50 msgr2=0x7f332c0b5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:31.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 shutdown_connections 2024-03-23T15:51:31.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:31.023+0000 7f333148f640 1 -- 172.21.15.129:0/73671609 wait complete. 2024-03-23T15:51:31.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:31.641 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:32.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:32.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.742+0000 7f5509720640 1 Processor -- start 2024-03-23T15:51:32.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.742+0000 7f5509720640 1 -- start start 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f5509720640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f550415a2e0 0x7f550406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f5509720640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55040cbcb0 con 0x7f550406c0b0 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f5509720640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f550415b280 con 0x7f55041544b0 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f5509720640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f55040c5be0 con 0x7f550415a2e0 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f55027fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f550415a2e0 0x7f550406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f5502ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f55041544b0 0x7f55040caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55438/0 (socket says 172.21.15.129:55438) 2024-03-23T15:51:32.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.743+0000 7f5502ffd640 1 -- 172.21.15.129:0/2233898123 learned_addr learned my addr 172.21.15.129:0/2233898123 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.746+0000 7f5501ffb640 1 -- 172.21.15.129:0/2233898123 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015731989 0 0) 0x7f550415b280 con 0x7f55041544b0 2024-03-23T15:51:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.746+0000 7f5501ffb640 1 -- 172.21.15.129:0/2233898123 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54ec003520 con 0x7f55041544b0 2024-03-23T15:51:32.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.747+0000 7f55027fc640 1 -- 172.21.15.129:0/2233898123 >> v1:172.21.15.157:6789/0 conn(0x7f55041544b0 legacy=0x7f55040caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:32.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.747+0000 7f55027fc640 1 -- 172.21.15.129:0/2233898123 >> v1:172.21.15.129:6789/0 conn(0x7f550406c0b0 legacy=0x7f55040c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:32.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.747+0000 7f55027fc640 1 -- 172.21.15.129:0/2233898123 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550415a9e0 con 0x7f550415a2e0 2024-03-23T15:51:32.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.753+0000 7f55027fc640 1 --2- 172.21.15.129:0/2233898123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550415a2e0 0x7f550406bb70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f54f80097a0 tx=0x7f54f802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e686dad1a4f833a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.757+0000 7f5501ffb640 1 -- 172.21.15.129:0/2233898123 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f802f810 con 0x7f550415a2e0 2024-03-23T15:51:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.757+0000 7f5501ffb640 1 -- 172.21.15.129:0/2233898123 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f802f970 con 0x7f550415a2e0 2024-03-23T15:51:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.757+0000 7f5501ffb640 1 -- 172.21.15.129:0/2233898123 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f802fd00 con 0x7f550415a2e0 2024-03-23T15:51:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 -- 172.21.15.129:0/2233898123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550415a2e0 msgr2=0x7f550406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 --2- 172.21.15.129:0/2233898123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550415a2e0 0x7f550406bb70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f54f80097a0 tx=0x7f54f802b9f0 comp rx=0 tx=0).stop 2024-03-23T15:51:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 -- 172.21.15.129:0/2233898123 shutdown_connections 2024-03-23T15:51:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 --2- 172.21.15.129:0/2233898123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550415a2e0 0x7f550406bb70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:32.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 -- 172.21.15.129:0/2233898123 >> 172.21.15.129:0/2233898123 conn(0x7f5504061b50 msgr2=0x7f5504061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:32.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 -- 172.21.15.129:0/2233898123 shutdown_connections 2024-03-23T15:51:32.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.758+0000 7f5509720640 1 -- 172.21.15.129:0/2233898123 wait complete. 2024-03-23T15:51:32.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 Processor -- start 2024-03-23T15:51:32.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 -- start start 2024-03-23T15:51:32.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550406c0b0 0x7f5504156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:32.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55041544b0 0x7f55040c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:32.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 0x7f55040c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5504155020 con 0x7f550406c0b0 2024-03-23T15:51:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5504151990 con 0x7f550415a2e0 2024-03-23T15:51:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.760+0000 7f5509720640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5504151be0 con 0x7f55041544b0 2024-03-23T15:51:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.761+0000 7f5502ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550406c0b0 0x7f5504156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.761+0000 7f55037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 0x7f55040c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.761+0000 7f55037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 0x7f55040c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40730/0 (socket says 172.21.15.129:40730) 2024-03-23T15:51:32.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.761+0000 7f55037fe640 1 -- 172.21.15.129:0/3535220093 learned_addr learned my addr 172.21.15.129:0/3535220093 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:32.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.761+0000 7f55027fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55041544b0 0x7f55040c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.766+0000 7f55037fe640 1 -- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55041544b0 msgr2=0x7f55040c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.766+0000 7f55037fe640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55041544b0 0x7f55040c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.766+0000 7f55037fe640 1 -- 172.21.15.129:0/3535220093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550406c0b0 msgr2=0x7f5504156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.766+0000 7f55037fe640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550406c0b0 0x7f5504156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.766+0000 7f55037fe640 1 -- 172.21.15.129:0/3535220093 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f8009400 con 0x7f550415a2e0 2024-03-23T15:51:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.766+0000 7f5502ffd640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550406c0b0 0x7f5504156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:51:32.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.769+0000 7f55037fe640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 0x7f55040c8840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00c7a0 tx=0x7f54fc00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:32.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.774+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc003b70 con 0x7f550415a2e0 2024-03-23T15:51:32.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.774+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54fc003cf0 con 0x7f550415a2e0 2024-03-23T15:51:32.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.774+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5504151ec0 con 0x7f550415a2e0 2024-03-23T15:51:32.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.774+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc003ea0 con 0x7f550415a2e0 2024-03-23T15:51:32.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.774+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55040c73f0 con 0x7f550415a2e0 2024-03-23T15:51:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.775+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5504152100 con 0x7f550415a2e0 2024-03-23T15:51:32.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.798+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f54fc00aa90 con 0x7f550415a2e0 2024-03-23T15:51:32.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.798+0000 7f54eb7fe640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f54f006b860 0x7f54f006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:32.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.799+0000 7f5502ffd640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f54f006b860 0x7f54f006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:32.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:32.799+0000 7f5502ffd640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f54f006b860 0x7f54f006dd20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f54f40011e0 tx=0x7f54f4007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:33.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.063+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f54fc0590b0 con 0x7f550415a2e0 2024-03-23T15:51:33.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.084+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f54fc055350 con 0x7f550415a2e0 2024-03-23T15:51:33.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.322+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5504154b10 con 0x7f550415a2e0 2024-03-23T15:51:33.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.346+0000 7f54eb7fe640 1 -- 172.21.15.129:0/3535220093 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5504154b10 con 0x7f550415a2e0 2024-03-23T15:51:33.349 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:33.350 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:33.350 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f54f006b860 msgr2=0x7f54f006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f54f006b860 0x7f54f006dd20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f54f40011e0 tx=0x7f54f4007d30 comp rx=0 tx=0).stop 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 msgr2=0x7f55040c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 0x7f55040c8840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00c7a0 tx=0x7f54fc00cc60 comp rx=0 tx=0).stop 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 shutdown_connections 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f54f006b860 0x7f54f006dd20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f550415a2e0 0x7f55040c8840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55041544b0 0x7f55040c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 --2- 172.21.15.129:0/3535220093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f550406c0b0 0x7f5504156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 >> 172.21.15.129:0/3535220093 conn(0x7f5504061b50 msgr2=0x7f5504152910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 shutdown_connections 2024-03-23T15:51:33.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:33.348+0000 7f5509720640 1 -- 172.21.15.129:0/3535220093 wait complete. 2024-03-23T15:51:33.968 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:33.968 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:34.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:35.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.069+0000 7f066b3c7640 1 Processor -- start 2024-03-23T15:51:35.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.069+0000 7f066b3c7640 1 -- start start 2024-03-23T15:51:35.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f066b3c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f066406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:35.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f066b3c7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06640cbcb0 con 0x7f066406c0b0 2024-03-23T15:51:35.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f066b3c7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f066415b280 con 0x7f06641544b0 2024-03-23T15:51:35.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f066b3c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06640c5be0 con 0x7f066415a2e0 2024-03-23T15:51:35.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f0669163640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f06641544b0 0x7f06640caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55444/0 (socket says 172.21.15.129:55444) 2024-03-23T15:51:35.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f0669163640 1 -- 172.21.15.129:0/1516907298 learned_addr learned my addr 172.21.15.129:0/1516907298 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:35.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.070+0000 7f0668962640 1 --2- 172.21.15.129:0/1516907298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f066406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:35.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.073+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055257651 0 0) 0x7f066415b280 con 0x7f06641544b0 2024-03-23T15:51:35.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.073+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0650003520 con 0x7f06641544b0 2024-03-23T15:51:35.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.073+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953135768 0 0) 0x7f06640cbcb0 con 0x7f066406c0b0 2024-03-23T15:51:35.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.073+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f066415b280 con 0x7f066406c0b0 2024-03-23T15:51:35.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.074+0000 7f0668962640 1 -- 172.21.15.129:0/1516907298 >> v1:172.21.15.157:6789/0 conn(0x7f06641544b0 legacy=0x7f06640caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.074+0000 7f0668962640 1 -- 172.21.15.129:0/1516907298 >> v1:172.21.15.129:6789/0 conn(0x7f066406c0b0 legacy=0x7f06640c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.074+0000 7f0668962640 1 -- 172.21.15.129:0/1516907298 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066415a9e0 con 0x7f066415a2e0 2024-03-23T15:51:35.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.077+0000 7f0668962640 1 --2- 172.21.15.129:0/1516907298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f066406bb70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f06540097d0 tx=0x7f065402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abbb1e2ee48b83ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:35.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.082+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654030880 con 0x7f066415a2e0 2024-03-23T15:51:35.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.082+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06540309e0 con 0x7f066415a2e0 2024-03-23T15:51:35.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.082+0000 7f065bfff640 1 -- 172.21.15.129:0/1516907298 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654030d70 con 0x7f066415a2e0 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 -- 172.21.15.129:0/1516907298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 msgr2=0x7f066406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 --2- 172.21.15.129:0/1516907298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f066406bb70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f06540097d0 tx=0x7f065402b9c0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 -- 172.21.15.129:0/1516907298 shutdown_connections 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 --2- 172.21.15.129:0/1516907298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f066406bb70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 -- 172.21.15.129:0/1516907298 >> 172.21.15.129:0/1516907298 conn(0x7f0664061b50 msgr2=0x7f0664061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 -- 172.21.15.129:0/1516907298 shutdown_connections 2024-03-23T15:51:35.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.083+0000 7f066b3c7640 1 -- 172.21.15.129:0/1516907298 wait complete. 2024-03-23T15:51:35.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.084+0000 7f066b3c7640 1 Processor -- start 2024-03-23T15:51:35.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.084+0000 7f066b3c7640 1 -- start start 2024-03-23T15:51:35.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f066b3c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 0x7f06640c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:35.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f066b3c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06641544b0 0x7f06641f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:35.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f066b3c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f0664146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:35.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f066b3c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06640c7f80 con 0x7f06641544b0 2024-03-23T15:51:35.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f066b3c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06640ce370 con 0x7f066406c0b0 2024-03-23T15:51:35.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f066b3c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06640c6fa0 con 0x7f066415a2e0 2024-03-23T15:51:35.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f0668962640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06641544b0 0x7f06641f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:35.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.085+0000 7f0669163640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 0x7f06640c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:35.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.086+0000 7f0669964640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f0664146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:35.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.086+0000 7f0669163640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 0x7f06640c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40736/0 (socket says 172.21.15.129:40736) 2024-03-23T15:51:35.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.086+0000 7f0669163640 1 -- 172.21.15.129:0/239778048 learned_addr learned my addr 172.21.15.129:0/239778048 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.091+0000 7f0669163640 1 -- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 msgr2=0x7f0664146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.091+0000 7f0669163640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f0664146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.091+0000 7f0669163640 1 -- 172.21.15.129:0/239778048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06641544b0 msgr2=0x7f06641f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.091+0000 7f0669163640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06641544b0 0x7f06641f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.091+0000 7f0669163640 1 -- 172.21.15.129:0/239778048 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0654009400 con 0x7f066406c0b0 2024-03-23T15:51:35.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.091+0000 7f0668962640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06641544b0 0x7f06641f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:51:35.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.094+0000 7f0669163640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 0x7f06640c7a40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f066000c480 tx=0x7f066000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:35.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.098+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660005ce0 con 0x7f066406c0b0 2024-03-23T15:51:35.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.098+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0660005e40 con 0x7f066406c0b0 2024-03-23T15:51:35.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.098+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06641f3a70 con 0x7f066406c0b0 2024-03-23T15:51:35.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.098+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06600072b0 con 0x7f066406c0b0 2024-03-23T15:51:35.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.099+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06640c7240 con 0x7f066406c0b0 2024-03-23T15:51:35.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.100+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0664075740 con 0x7f066406c0b0 2024-03-23T15:51:35.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.129+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0660007410 con 0x7f066406c0b0 2024-03-23T15:51:35.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.129+0000 7f0659ffb640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f064806b890 0x7f064806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:35.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.130+0000 7f0668962640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f064806b890 0x7f064806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:35.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.130+0000 7f0668962640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f064806b890 0x7f064806dd50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0654036000 tx=0x7f065403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:35.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.136+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f06600585a0 con 0x7f066406c0b0 2024-03-23T15:51:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.394+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0660054840 con 0x7f066406c0b0 2024-03-23T15:51:35.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.617+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0664157d00 con 0x7f066406c0b0 2024-03-23T15:51:35.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.658+0000 7f0659ffb640 1 -- 172.21.15.129:0/239778048 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0660017090 con 0x7f066406c0b0 2024-03-23T15:51:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:35.662 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f064806b890 msgr2=0x7f064806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f064806b890 0x7f064806dd50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0654036000 tx=0x7f065403b040 comp rx=0 tx=0).stop 2024-03-23T15:51:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 msgr2=0x7f06640c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 0x7f06640c7a40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f066000c480 tx=0x7f066000c940 comp rx=0 tx=0).stop 2024-03-23T15:51:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 shutdown_connections 2024-03-23T15:51:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f066415a2e0 0x7f0664146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06641544b0 0x7f06641f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f064806b890 0x7f064806dd50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 --2- 172.21.15.129:0/239778048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f066406c0b0 0x7f06640c7a40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:35.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 >> 172.21.15.129:0/239778048 conn(0x7f0664061b50 msgr2=0x7f06641528f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:35.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 shutdown_connections 2024-03-23T15:51:35.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:35.661+0000 7f066b3c7640 1 -- 172.21.15.129:0/239778048 wait complete. 2024-03-23T15:51:36.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:36.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:36.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:51:36.927 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:51:37.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:37.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.379+0000 7f52ccd9a640 1 Processor -- start 2024-03-23T15:51:37.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.379+0000 7f52ccd9a640 1 -- start start 2024-03-23T15:51:37.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.380+0000 7f52ccd9a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f52c806c030 0x7f52c806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:37.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.380+0000 7f52ccd9a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f52c80cc080 0x7f52c815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:37.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.380+0000 7f52ccd9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c806cae0 con 0x7f52c80cbc60 2024-03-23T15:51:37.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.380+0000 7f52ccd9a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f52c80cabc0 con 0x7f52c80cc080 2024-03-23T15:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.380+0000 7f52ccd9a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f52c80caea0 con 0x7f52c806c030 2024-03-23T15:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.381+0000 7f52c6d9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f52c80cc080 0x7f52c815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.381+0000 7f52c6d9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f52c80cc080 0x7f52c815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51600/0 (socket says 172.21.15.129:51600) 2024-03-23T15:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.381+0000 7f52c659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f52c806c030 0x7f52c806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.381+0000 7f52c5d9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f52c80cbc60 0x7f52c815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:55446/0 (socket says 172.21.15.129:55446) 2024-03-23T15:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.381+0000 7f52c6d9d640 1 -- 172.21.15.129:0/3347302888 learned_addr learned my addr 172.21.15.129:0/3347302888 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.383+0000 7f52c6d9d640 1 -- 172.21.15.129:0/3347302888 >> v2:172.21.15.157:3300/0 conn(0x7f52c806c030 msgr2=0x7f52c806c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.383+0000 7f52c6d9d640 1 --2- 172.21.15.129:0/3347302888 >> v2:172.21.15.157:3300/0 conn(0x7f52c806c030 0x7f52c806c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.383+0000 7f52c6d9d640 1 -- 172.21.15.129:0/3347302888 >> v1:172.21.15.157:6789/0 conn(0x7f52c80cbc60 legacy=0x7f52c815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.383+0000 7f52c6d9d640 1 -- 172.21.15.129:0/3347302888 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c80cb180 con 0x7f52c80cc080 2024-03-23T15:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.385+0000 7f52c6d9d640 1 --2- 172.21.15.129:0/3347302888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c815e010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52c80cf770 tx=0x7f52c002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6272d5693891620 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.389+0000 7f52c559a640 1 -- 172.21.15.129:0/3347302888 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c002cb20 con 0x7f52c80cc080 2024-03-23T15:51:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.389+0000 7f52c559a640 1 -- 172.21.15.129:0/3347302888 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c002cc80 con 0x7f52c80cc080 2024-03-23T15:51:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.389+0000 7f52c559a640 1 -- 172.21.15.129:0/3347302888 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c00343f0 con 0x7f52c80cc080 2024-03-23T15:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 -- 172.21.15.129:0/3347302888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 msgr2=0x7f52c815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/3347302888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c815e010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52c80cf770 tx=0x7f52c002bf80 comp rx=0 tx=0).stop 2024-03-23T15:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 -- 172.21.15.129:0/3347302888 shutdown_connections 2024-03-23T15:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/3347302888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c815e010 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/3347302888 >> v2:172.21.15.157:3300/0 conn(0x7f52c806c030 0x7f52c806c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 -- 172.21.15.129:0/3347302888 >> 172.21.15.129:0/3347302888 conn(0x7f52c80cfb20 msgr2=0x7f52c80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 -- 172.21.15.129:0/3347302888 shutdown_connections 2024-03-23T15:51:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.390+0000 7f52ccd9a640 1 -- 172.21.15.129:0/3347302888 wait complete. 2024-03-23T15:51:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.391+0000 7f52ccd9a640 1 Processor -- start 2024-03-23T15:51:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.391+0000 7f52ccd9a640 1 -- start start 2024-03-23T15:51:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.392+0000 7f52ccd9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 0x7f52c815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.392+0000 7f52ccd9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52c80cbc60 0x7f52c8155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.392+0000 7f52ccd9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c8155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.392+0000 7f52ccd9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52c8154d90 con 0x7f52c80cc080 2024-03-23T15:51:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.392+0000 7f52ccd9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f52c8156b30 con 0x7f52c806c030 2024-03-23T15:51:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.392+0000 7f52ccd9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f52c81574c0 con 0x7f52c80cbc60 2024-03-23T15:51:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.393+0000 7f52c6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c8155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.393+0000 7f52c659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 0x7f52c815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.393+0000 7f52c659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 0x7f52c815d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40752/0 (socket says 172.21.15.129:40752) 2024-03-23T15:51:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.393+0000 7f52c659c640 1 -- 172.21.15.129:0/2911990418 learned_addr learned my addr 172.21.15.129:0/2911990418 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.393+0000 7f52c5d9b640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52c80cbc60 0x7f52c8155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:37.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.397+0000 7f52c659c640 1 -- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52c80cbc60 msgr2=0x7f52c8155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:37.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.397+0000 7f52c659c640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52c80cbc60 0x7f52c8155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.397+0000 7f52c659c640 1 -- 172.21.15.129:0/2911990418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 msgr2=0x7f52c8155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:37.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.397+0000 7f52c659c640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c8155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.397+0000 7f52c659c640 1 -- 172.21.15.129:0/2911990418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c0009b30 con 0x7f52c806c030 2024-03-23T15:51:37.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.400+0000 7f52c659c640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 0x7f52c815d510 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f52b800e9b0 tx=0x7f52b800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.405+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8009670 con 0x7f52c806c030 2024-03-23T15:51:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.405+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b800c9a0 con 0x7f52c806c030 2024-03-23T15:51:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.405+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b800cd50 con 0x7f52c806c030 2024-03-23T15:51:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.405+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c81f0a00 con 0x7f52c806c030 2024-03-23T15:51:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.405+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c81570b0 con 0x7f52c806c030 2024-03-23T15:51:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.406+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c8075740 con 0x7f52c806c030 2024-03-23T15:51:37.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.430+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f52b8013470 con 0x7f52c806c030 2024-03-23T15:51:37.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.430+0000 7f52b6ffd640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f52a806b810 0x7f52a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:37.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.430+0000 7f52c5d9b640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f52a806b810 0x7f52a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:37.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.431+0000 7f52c5d9b640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f52a806b810 0x7f52a806dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f52c81562f0 tx=0x7f52bc007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:37.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.693+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f52b8014e40 con 0x7f52c806c030 2024-03-23T15:51:37.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.704+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f52b8055680 con 0x7f52c806c030 2024-03-23T15:51:37.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.950+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52c80d0a30 con 0x7f52c806c030 2024-03-23T15:51:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.962+0000 7f52b6ffd640 1 -- 172.21.15.129:0/2911990418 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f52b8017020 con 0x7f52c806c030 2024-03-23T15:51:37.966 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:37.966 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:37.966 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:37.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f52a806b810 msgr2=0x7f52a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:37.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f52a806b810 0x7f52a806dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f52c81562f0 tx=0x7f52bc007e80 comp rx=0 tx=0).stop 2024-03-23T15:51:37.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 msgr2=0x7f52c815d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:37.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 0x7f52c815d510 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f52b800e9b0 tx=0x7f52b800ee70 comp rx=0 tx=0).stop 2024-03-23T15:51:37.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 shutdown_connections 2024-03-23T15:51:37.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52c80cc080 0x7f52c8155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52c80cbc60 0x7f52c8155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f52a806b810 0x7f52a806dcd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 --2- 172.21.15.129:0/2911990418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52c806c030 0x7f52c815d510 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:37.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 >> 172.21.15.129:0/2911990418 conn(0x7f52c80cfb20 msgr2=0x7f52c80ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:37.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 shutdown_connections 2024-03-23T15:51:37.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:37.965+0000 7f52ccd9a640 1 -- 172.21.15.129:0/2911990418 wait complete. 2024-03-23T15:51:38.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:38.583 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:39.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.689+0000 7faa0753e640 1 Processor -- start 2024-03-23T15:51:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.689+0000 7faa0753e640 1 -- start start 2024-03-23T15:51:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.690+0000 7faa0753e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faa001544b0 0x7faa0015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.690+0000 7faa0753e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000cbcb0 con 0x7faa000c6000 2024-03-23T15:51:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.690+0000 7faa0753e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0015b280 con 0x7faa000caa00 2024-03-23T15:51:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.690+0000 7faa0753e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faa000c5be0 con 0x7faa001544b0 2024-03-23T15:51:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.691+0000 7faa05adb640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7faa000c6000 0x7faa00164c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:42528/0 (socket says 172.21.15.129:42528) 2024-03-23T15:51:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.691+0000 7faa05adb640 1 -- 172.21.15.129:0/3009650704 learned_addr learned my addr 172.21.15.129:0/3009650704 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.691+0000 7faa052da640 1 --2- 172.21.15.129:0/3009650704 >> v2:172.21.15.157:3300/0 conn(0x7faa001544b0 0x7faa0015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:39.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.694+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247752703 0 0) 0x7faa000cbcb0 con 0x7faa000c6000 2024-03-23T15:51:39.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.694+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9ec003520 con 0x7faa000c6000 2024-03-23T15:51:39.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.696+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393963397 0 0) 0x7faa0015b280 con 0x7faa000caa00 2024-03-23T15:51:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.696+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa000cbcb0 con 0x7faa000caa00 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1597901748 0 0) 0x7fa9ec003520 con 0x7faa000c6000 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 >> v2:172.21.15.157:3300/0 conn(0x7faa001544b0 msgr2=0x7faa0015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 --2- 172.21.15.129:0/3009650704 >> v2:172.21.15.157:3300/0 conn(0x7faa001544b0 0x7faa0015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 >> v1:172.21.15.157:6789/0 conn(0x7faa000caa00 legacy=0x7faa0006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0015a9e0 con 0x7faa000c6000 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7fa9f80019f0 con 0x7faa000c6000 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 >> v1:172.21.15.129:6789/0 conn(0x7faa000c6000 legacy=0x7faa00164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9ec003b50 0x7fa9ec003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9ec005180 0x7fa9ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa000c5be0 con 0x7fa9ec0047c0 2024-03-23T15:51:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec00bfa0 con 0x7fa9ec005180 2024-03-23T15:51:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.697+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ec00c160 con 0x7fa9ec003b50 2024-03-23T15:51:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.698+0000 7faa04ad9640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.698+0000 7faa052da640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9ec003b50 0x7fa9ec003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:39.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.702+0000 7faa052da640 1 -- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9ec005180 msgr2=0x7fa9ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.702+0000 7faa052da640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9ec005180 0x7fa9ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.702+0000 7faa052da640 1 -- 172.21.15.129:0/3009650704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec0047c0 msgr2=0x7fa9ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.702+0000 7faa052da640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.702+0000 7faa052da640 1 -- 172.21.15.129:0/3009650704 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec003520 con 0x7fa9ec003b50 2024-03-23T15:51:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.702+0000 7faa05adb640 1 -- 172.21.15.129:0/3009650704 reap_dead start 2024-03-23T15:51:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.705+0000 7faa052da640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9ec003b50 0x7fa9ec003f50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009ac0 tx=0x7fa9ec0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50b7a723cf9f15cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.709+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc02c6d0 con 0x7fa9ec003b50 2024-03-23T15:51:39.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.709+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9fc02c830 con 0x7fa9ec003b50 2024-03-23T15:51:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.709+0000 7fa9f7fff640 1 -- 172.21.15.129:0/3009650704 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc02cbc0 con 0x7fa9ec003b50 2024-03-23T15:51:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 -- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9ec003b50 msgr2=0x7fa9ec003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9ec003b50 0x7fa9ec003f50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009ac0 tx=0x7fa9ec0057f0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 -- 172.21.15.129:0/3009650704 shutdown_connections 2024-03-23T15:51:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 --2- 172.21.15.129:0/3009650704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9ec003b50 0x7fa9ec003f50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 -- 172.21.15.129:0/3009650704 >> 172.21.15.129:0/3009650704 conn(0x7faa00061b50 msgr2=0x7faa00061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 -- 172.21.15.129:0/3009650704 shutdown_connections 2024-03-23T15:51:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.710+0000 7faa0753e640 1 -- 172.21.15.129:0/3009650704 wait complete. 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.711+0000 7faa0753e640 1 Processor -- start 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.711+0000 7faa0753e640 1 -- start start 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.712+0000 7faa0753e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 0x7faa00151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.712+0000 7faa0753e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa001f0780 0x7faa00152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.712+0000 7faa0753e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa00155070 0x7faa00155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.712+0000 7faa0753e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa00151910 con 0x7fa9ec003b50 2024-03-23T15:51:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.712+0000 7faa0753e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faa000c70a0 con 0x7faa001f0780 2024-03-23T15:51:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.712+0000 7faa0753e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faa000c7380 con 0x7faa00155070 2024-03-23T15:51:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.713+0000 7faa052da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 0x7faa00151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.713+0000 7faa05adb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa00155070 0x7faa00155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.713+0000 7faa052da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 0x7faa00151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51638/0 (socket says 172.21.15.129:51638) 2024-03-23T15:51:39.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.713+0000 7faa052da640 1 -- 172.21.15.129:0/669916271 learned_addr learned my addr 172.21.15.129:0/669916271 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.717+0000 7faa052da640 1 -- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa00155070 msgr2=0x7faa00155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.717+0000 7faa052da640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa00155070 0x7faa00155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.717+0000 7faa052da640 1 -- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa001f0780 msgr2=0x7faa00152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.717+0000 7faa052da640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa001f0780 0x7faa00152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.717+0000 7faa052da640 1 -- 172.21.15.129:0/669916271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa000c7660 con 0x7fa9ec003b50 2024-03-23T15:51:39.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.719+0000 7faa052da640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 0x7faa00151280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009be0 tx=0x7fa9fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.723+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc003810 con 0x7fa9ec003b50 2024-03-23T15:51:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.723+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9fc003970 con 0x7fa9ec003b50 2024-03-23T15:51:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.723+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa000c7940 con 0x7fa9ec003b50 2024-03-23T15:51:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.723+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc003d00 con 0x7fa9ec003b50 2024-03-23T15:51:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.723+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa001f0bc0 con 0x7fa9ec003b50 2024-03-23T15:51:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.724+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa00075740 con 0x7fa9ec003b50 2024-03-23T15:51:39.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.976+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa9fc03b070 con 0x7fa9ec003b50 2024-03-23T15:51:39.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.977+0000 7fa9f5ffb640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa9e406b890 0x7fa9e406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:39.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.977+0000 7faa04ad9640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa9e406b890 0x7fa9e406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:39.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.978+0000 7faa04ad9640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa9e406b890 0x7fa9e406dd50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00079a0 tx=0x7fa9f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:39.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.991+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa9fc07abc0 con 0x7fa9ec003b50 2024-03-23T15:51:39.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:39.991+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9fc076e60 con 0x7fa9ec003b50 2024-03-23T15:51:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.238+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa00151d20 con 0x7fa9ec003b50 2024-03-23T15:51:40.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.253+0000 7fa9f5ffb640 1 -- 172.21.15.129:0/669916271 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa9fc04c020 con 0x7fa9ec003b50 2024-03-23T15:51:40.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:40.257 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:40.257 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.255+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa9e406b890 msgr2=0x7fa9e406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.255+0000 7faa0753e640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa9e406b890 0x7fa9e406dd50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00079a0 tx=0x7fa9f0009040 comp rx=0 tx=0).stop 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.255+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 msgr2=0x7faa00151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.255+0000 7faa0753e640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 0x7faa00151280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009be0 tx=0x7fa9fc004730 comp rx=0 tx=0).stop 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 shutdown_connections 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa00155070 0x7faa00155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa9e406b890 0x7fa9e406dd50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa001f0780 0x7faa00152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 --2- 172.21.15.129:0/669916271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec003b50 0x7faa00151280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 >> 172.21.15.129:0/669916271 conn(0x7faa00061b50 msgr2=0x7faa0015a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 shutdown_connections 2024-03-23T15:51:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:40.256+0000 7faa0753e640 1 -- 172.21.15.129:0/669916271 wait complete. 2024-03-23T15:51:40.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:40.879 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:41.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:41.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.979+0000 7f9566793640 1 Processor -- start 2024-03-23T15:51:41.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.979+0000 7f9566793640 1 -- start start 2024-03-23T15:51:41.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.980+0000 7f9566793640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f956015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:41.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.980+0000 7f9566793640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f95600cef40 0x7f956015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:41.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.980+0000 7f9566793640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f956015b290 con 0x7f95600ccd60 2024-03-23T15:51:41.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.980+0000 7f9566793640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f95600ce800 con 0x7f95600cef40 2024-03-23T15:51:41.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.980+0000 7f9566793640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9560154a70 con 0x7f95600ceb00 2024-03-23T15:51:41.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.981+0000 7f9564d30640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f95600cef40 0x7f956015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:41.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.981+0000 7f955ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f95600ccd60 0x7f956006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42610/0 (socket says 172.21.15.129:42610) 2024-03-23T15:51:41.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.981+0000 7f955ffff640 1 -- 172.21.15.129:0/2518830725 learned_addr learned my addr 172.21.15.129:0/2518830725 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:41.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.981+0000 7f955f7fe640 1 --2- 172.21.15.129:0/2518830725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f956015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:41.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.983+0000 7f955f7fe640 1 -- 172.21.15.129:0/2518830725 >> v2:172.21.15.129:3300/0 conn(0x7f95600cef40 msgr2=0x7f956015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:41.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.983+0000 7f955f7fe640 1 --2- 172.21.15.129:0/2518830725 >> v2:172.21.15.129:3300/0 conn(0x7f95600cef40 0x7f956015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:41.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.983+0000 7f955f7fe640 1 -- 172.21.15.129:0/2518830725 >> v1:172.21.15.157:6789/0 conn(0x7f95600ccd60 legacy=0x7f956006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:41.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.983+0000 7f955f7fe640 1 -- 172.21.15.129:0/2518830725 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95600cd950 con 0x7f95600ceb00 2024-03-23T15:51:41.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.985+0000 7f955f7fe640 1 --2- 172.21.15.129:0/2518830725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f956015ad50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f95540097a0 tx=0x7f955402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee4dfb54ca55bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.990+0000 7f955effd640 1 -- 172.21.15.129:0/2518830725 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554030800 con 0x7f95600ceb00 2024-03-23T15:51:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.990+0000 7f955effd640 1 -- 172.21.15.129:0/2518830725 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9554030960 con 0x7f95600ceb00 2024-03-23T15:51:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.990+0000 7f955effd640 1 -- 172.21.15.129:0/2518830725 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554030cf0 con 0x7f95600ceb00 2024-03-23T15:51:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 -- 172.21.15.129:0/2518830725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 msgr2=0x7f956015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 --2- 172.21.15.129:0/2518830725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f956015ad50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f95540097a0 tx=0x7f955402b940 comp rx=0 tx=0).stop 2024-03-23T15:51:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 -- 172.21.15.129:0/2518830725 shutdown_connections 2024-03-23T15:51:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 --2- 172.21.15.129:0/2518830725 >> v2:172.21.15.129:3300/0 conn(0x7f95600cef40 0x7f956015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 --2- 172.21.15.129:0/2518830725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f956015ad50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 -- 172.21.15.129:0/2518830725 >> 172.21.15.129:0/2518830725 conn(0x7f95600b1840 msgr2=0x7f95600b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:41.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 -- 172.21.15.129:0/2518830725 shutdown_connections 2024-03-23T15:51:41.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.991+0000 7f9566793640 1 -- 172.21.15.129:0/2518830725 wait complete. 2024-03-23T15:51:41.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.993+0000 7f9566793640 1 Processor -- start 2024-03-23T15:51:41.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.993+0000 7f9566793640 1 -- start start 2024-03-23T15:51:41.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.994+0000 7f9566793640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95600ccd60 0x7f956015c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:41.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.994+0000 7f9566793640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f95600c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:41.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.994+0000 7f9566793640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95600cef40 0x7f95601e43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.994+0000 7f9566793640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95601e5a10 con 0x7f95600cef40 2024-03-23T15:51:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.994+0000 7f9566793640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f956014b9e0 con 0x7f95600ccd60 2024-03-23T15:51:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.994+0000 7f9566793640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95601e3640 con 0x7f95600ceb00 2024-03-23T15:51:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.995+0000 7f955ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95600ccd60 0x7f956015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.995+0000 7f9564d30640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95600cef40 0x7f95601e43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.995+0000 7f955f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f95600c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.995+0000 7f9564d30640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95600cef40 0x7f95601e43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53326/0 (socket says 172.21.15.129:53326) 2024-03-23T15:51:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.995+0000 7f9564d30640 1 -- 172.21.15.129:0/2639551356 learned_addr learned my addr 172.21.15.129:0/2639551356 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.999+0000 7f955f7fe640 1 -- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95600ccd60 msgr2=0x7f956015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.999+0000 7f955f7fe640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95600ccd60 0x7f956015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.999+0000 7f955f7fe640 1 -- 172.21.15.129:0/2639551356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95600cef40 msgr2=0x7f95601e43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.999+0000 7f955f7fe640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95600cef40 0x7f95601e43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:41.999+0000 7f955f7fe640 1 -- 172.21.15.129:0/2639551356 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9554009400 con 0x7f95600ceb00 2024-03-23T15:51:42.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.002+0000 7f955f7fe640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f95600c9770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9560154bb0 tx=0x7f9554004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.007+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955403e040 con 0x7f95600ceb00 2024-03-23T15:51:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.007+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9554039cb0 con 0x7f95600ceb00 2024-03-23T15:51:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.007+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95600c9cb0 con 0x7f95600ceb00 2024-03-23T15:51:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.007+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554043680 con 0x7f95600ceb00 2024-03-23T15:51:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.007+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f956015d670 con 0x7f95600ceb00 2024-03-23T15:51:42.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.008+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9540005290 con 0x7f95600ceb00 2024-03-23T15:51:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.285+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f955404c3b0 con 0x7f95600ceb00 2024-03-23T15:51:42.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.286+0000 7f955cff9640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f954c06b810 0x7f954c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:42.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.286+0000 7f955ffff640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f954c06b810 0x7f954c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:42.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.286+0000 7f955ffff640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f954c06b810 0x7f954c06dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9550007fd0 tx=0x7f9550007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:42.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.290+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9554039e20 con 0x7f95600ceb00 2024-03-23T15:51:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.301+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f95540772c0 con 0x7f95600ceb00 2024-03-23T15:51:42.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.517+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95400058f0 con 0x7f95600ceb00 2024-03-23T15:51:42.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.530+0000 7f955cff9640 1 -- 172.21.15.129:0/2639551356 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f95400058f0 con 0x7f95600ceb00 2024-03-23T15:51:42.534 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:42.534 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:42.534 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f954c06b810 msgr2=0x7f954c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f954c06b810 0x7f954c06dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9550007fd0 tx=0x7f9550007e80 comp rx=0 tx=0).stop 2024-03-23T15:51:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 msgr2=0x7f95600c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f95600c9770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9560154bb0 tx=0x7f9554004730 comp rx=0 tx=0).stop 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 shutdown_connections 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95600cef40 0x7f95601e43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95600ceb00 0x7f95600c9770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f954c06b810 0x7f954c06dcd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 --2- 172.21.15.129:0/2639551356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95600ccd60 0x7f956015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 >> 172.21.15.129:0/2639551356 conn(0x7f95600b1840 msgr2=0x7f95600b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 shutdown_connections 2024-03-23T15:51:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:42.533+0000 7f9566793640 1 -- 172.21.15.129:0/2639551356 wait complete. 2024-03-23T15:51:43.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:43.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:44.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:44.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.255+0000 7fbc37d3e640 1 Processor -- start 2024-03-23T15:51:44.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.255+0000 7fbc37d3e640 1 -- start start 2024-03-23T15:51:44.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.256+0000 7fbc37d3e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbc3015dba0 0x7fbc3006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:44.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.256+0000 7fbc37d3e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015df80 0x7fbc301540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:44.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.256+0000 7fbc37d3e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc30070640 con 0x7fbc300cca30 2024-03-23T15:51:44.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.256+0000 7fbc37d3e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fbc300ccea0 con 0x7fbc3015dba0 2024-03-23T15:51:44.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.256+0000 7fbc37d3e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbc300c4c00 con 0x7fbc3015df80 2024-03-23T15:51:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.256+0000 7fbc35ada640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbc3015dba0 0x7fbc3006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.257+0000 7fbc362db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015df80 0x7fbc301540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.257+0000 7fbc35ada640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fbc3015dba0 0x7fbc3006cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53340/0 (socket says 172.21.15.129:53340) 2024-03-23T15:51:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.257+0000 7fbc35ada640 1 -- 172.21.15.129:0/2595317522 learned_addr learned my addr 172.21.15.129:0/2595317522 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:44.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.260+0000 7fbc35ada640 1 -- 172.21.15.129:0/2595317522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015df80 msgr2=0x7fbc301540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:44.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.260+0000 7fbc35ada640 1 --2- 172.21.15.129:0/2595317522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015df80 0x7fbc301540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.260+0000 7fbc35ada640 1 -- 172.21.15.129:0/2595317522 >> v1:172.21.15.129:6789/0 conn(0x7fbc300cca30 legacy=0x7fbc3006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:44.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.260+0000 7fbc35ada640 1 -- 172.21.15.129:0/2595317522 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3006c240 con 0x7fbc3015dba0 2024-03-23T15:51:44.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.266+0000 7fbc35ada640 1 --2- 172.21.15.129:0/2595317522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3015dba0 0x7fbc3006cf70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0098a0 tx=0x7fbc2c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8560f188a9a0ff50 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:44.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc34ad8640 1 -- 172.21.15.129:0/2595317522 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c02f830 con 0x7fbc3015dba0 2024-03-23T15:51:44.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc34ad8640 1 -- 172.21.15.129:0/2595317522 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc2c02f990 con 0x7fbc3015dba0 2024-03-23T15:51:44.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc34ad8640 1 -- 172.21.15.129:0/2595317522 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c02fd20 con 0x7fbc3015dba0 2024-03-23T15:51:44.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 -- 172.21.15.129:0/2595317522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3015dba0 msgr2=0x7fbc3006cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/2595317522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3015dba0 0x7fbc3006cf70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0098a0 tx=0x7fbc2c02ba10 comp rx=0 tx=0).stop 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 -- 172.21.15.129:0/2595317522 shutdown_connections 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/2595317522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015df80 0x7fbc301540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/2595317522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc3015dba0 0x7fbc3006cf70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 -- 172.21.15.129:0/2595317522 >> 172.21.15.129:0/2595317522 conn(0x7fbc30071300 msgr2=0x7fbc300716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 -- 172.21.15.129:0/2595317522 shutdown_connections 2024-03-23T15:51:44.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.271+0000 7fbc37d3e640 1 -- 172.21.15.129:0/2595317522 wait complete. 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 Processor -- start 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 -- start start 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc300cca30 0x7fbc30071950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015dba0 0x7fbc301f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc3015df80 0x7fbc301f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc301f0730 con 0x7fbc300cca30 2024-03-23T15:51:44.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc301566a0 con 0x7fbc3015df80 2024-03-23T15:51:44.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.273+0000 7fbc37d3e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbc301f0c20 con 0x7fbc3015dba0 2024-03-23T15:51:44.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.274+0000 7fbc35ada640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc300cca30 0x7fbc30071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:44.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.274+0000 7fbc352d9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015dba0 0x7fbc301f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:44.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.274+0000 7fbc362db640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc3015df80 0x7fbc301f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:44.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.274+0000 7fbc35ada640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc300cca30 0x7fbc30071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53356/0 (socket says 172.21.15.129:53356) 2024-03-23T15:51:44.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.274+0000 7fbc35ada640 1 -- 172.21.15.129:0/1928471652 learned_addr learned my addr 172.21.15.129:0/1928471652 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:44.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.278+0000 7fbc352d9640 1 -- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc3015df80 msgr2=0x7fbc301f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:44.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.278+0000 7fbc352d9640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc3015df80 0x7fbc301f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.278+0000 7fbc352d9640 1 -- 172.21.15.129:0/1928471652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc300cca30 msgr2=0x7fbc30071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:44.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.278+0000 7fbc352d9640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc300cca30 0x7fbc30071950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.278+0000 7fbc352d9640 1 -- 172.21.15.129:0/1928471652 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c0094d0 con 0x7fbc3015dba0 2024-03-23T15:51:44.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.281+0000 7fbc352d9640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015dba0 0x7fbc301f1120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c3f0 tx=0x7fbc2000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:44.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.286+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200035f0 con 0x7fbc3015dba0 2024-03-23T15:51:44.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.286+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc20003750 con 0x7fbc3015dba0 2024-03-23T15:51:44.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.286+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20003b00 con 0x7fbc3015dba0 2024-03-23T15:51:44.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.286+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc30155db0 con 0x7fbc3015dba0 2024-03-23T15:51:44.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.286+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc30156050 con 0x7fbc3015dba0 2024-03-23T15:51:44.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.287+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc300b2ec0 con 0x7fbc3015dba0 2024-03-23T15:51:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.569+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc20003c60 con 0x7fbc3015dba0 2024-03-23T15:51:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.569+0000 7fbc267fc640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc1406b860 0x7fbc1406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.570+0000 7fbc35ada640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc1406b860 0x7fbc1406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.570+0000 7fbc35ada640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc1406b860 0x7fbc1406dd20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c02ba50 tx=0x7fbc2c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:44.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.573+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fbc20058350 con 0x7fbc3015dba0 2024-03-23T15:51:44.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.589+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc200545f0 con 0x7fbc3015dba0 2024-03-23T15:51:44.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.816+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc30075740 con 0x7fbc3015dba0 2024-03-23T15:51:44.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.897+0000 7fbc267fc640 1 -- 172.21.15.129:0/1928471652 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fbc20017090 con 0x7fbc3015dba0 2024-03-23T15:51:44.900 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:44.901 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:44.901 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.899+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc1406b860 msgr2=0x7fbc1406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.899+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc1406b860 0x7fbc1406dd20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c02ba50 tx=0x7fbc2c008030 comp rx=0 tx=0).stop 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.899+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015dba0 msgr2=0x7fbc301f1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.899+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015dba0 0x7fbc301f1120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c3f0 tx=0x7fbc2000c8b0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.899+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 shutdown_connections 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbc1406b860 0x7fbc1406dd20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc3015df80 0x7fbc301f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc3015dba0 0x7fbc301f1120 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 --2- 172.21.15.129:0/1928471652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc300cca30 0x7fbc30071950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 >> 172.21.15.129:0/1928471652 conn(0x7fbc30071300 msgr2=0x7fbc300ccfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 shutdown_connections 2024-03-23T15:51:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:44.900+0000 7fbc37d3e640 1 -- 172.21.15.129:0/1928471652 wait complete. 2024-03-23T15:51:45.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:45.518 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:46.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.619+0000 7f07317fd640 1 Processor -- start 2024-03-23T15:51:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.619+0000 7f07317fd640 1 -- start start 2024-03-23T15:51:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.620+0000 7f07317fd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f072c1544b0 0x7f072c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.620+0000 7f07317fd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.620+0000 7f07317fd640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f072c0cbcb0 con 0x7f072c06c0b0 2024-03-23T15:51:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.620+0000 7f07317fd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f072c0c5be0 con 0x7f072c1544b0 2024-03-23T15:51:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.620+0000 7f07317fd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f072c15b280 con 0x7f072c0caa00 2024-03-23T15:51:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.621+0000 7f072b7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f072c06c0b0 0x7f072c0c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43584/0 (socket says 172.21.15.129:43584) 2024-03-23T15:51:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.621+0000 7f072b7fe640 1 -- 172.21.15.129:0/2908750995 learned_addr learned my addr 172.21.15.129:0/2908750995 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.621+0000 7f072a7fc640 1 --2- 172.21.15.129:0/2908750995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.621+0000 7f072affd640 1 --2- 172.21.15.129:0/2908750995 >> v2:172.21.15.157:3300/0 conn(0x7f072c1544b0 0x7f072c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:46.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.624+0000 7f072a7fc640 1 -- 172.21.15.129:0/2908750995 >> v2:172.21.15.157:3300/0 conn(0x7f072c1544b0 msgr2=0x7f072c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:46.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.624+0000 7f072a7fc640 1 --2- 172.21.15.129:0/2908750995 >> v2:172.21.15.157:3300/0 conn(0x7f072c1544b0 0x7f072c15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:46.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.624+0000 7f072a7fc640 1 -- 172.21.15.129:0/2908750995 >> v1:172.21.15.129:6789/0 conn(0x7f072c06c0b0 legacy=0x7f072c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:46.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.624+0000 7f072a7fc640 1 -- 172.21.15.129:0/2908750995 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c15a9e0 con 0x7f072c0caa00 2024-03-23T15:51:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.626+0000 7f072a7fc640 1 --2- 172.21.15.129:0/2908750995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c06bb70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07200097a0 tx=0x7f072002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc3a10a81f52df11 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.630+0000 7f0729ffb640 1 -- 172.21.15.129:0/2908750995 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720030610 con 0x7f072c0caa00 2024-03-23T15:51:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.630+0000 7f0729ffb640 1 -- 172.21.15.129:0/2908750995 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0720030770 con 0x7f072c0caa00 2024-03-23T15:51:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.630+0000 7f0729ffb640 1 -- 172.21.15.129:0/2908750995 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720030b00 con 0x7f072c0caa00 2024-03-23T15:51:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 -- 172.21.15.129:0/2908750995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 msgr2=0x7f072c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 --2- 172.21.15.129:0/2908750995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c06bb70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07200097a0 tx=0x7f072002b960 comp rx=0 tx=0).stop 2024-03-23T15:51:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 -- 172.21.15.129:0/2908750995 shutdown_connections 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 --2- 172.21.15.129:0/2908750995 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c06bb70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 --2- 172.21.15.129:0/2908750995 >> v2:172.21.15.157:3300/0 conn(0x7f072c1544b0 0x7f072c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 -- 172.21.15.129:0/2908750995 >> 172.21.15.129:0/2908750995 conn(0x7f072c061b50 msgr2=0x7f072c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 -- 172.21.15.129:0/2908750995 shutdown_connections 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.631+0000 7f07317fd640 1 -- 172.21.15.129:0/2908750995 wait complete. 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 Processor -- start 2024-03-23T15:51:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 -- start start 2024-03-23T15:51:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c06c0b0 0x7f072c1f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c0c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c1544b0 0x7f072c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f072c156e00 con 0x7f072c06c0b0 2024-03-23T15:51:46.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1f3a10 con 0x7f072c1544b0 2024-03-23T15:51:46.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.633+0000 7f07317fd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f072c1567d0 con 0x7f072c0caa00 2024-03-23T15:51:46.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.634+0000 7f072affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c06c0b0 0x7f072c1f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:46.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.634+0000 7f072b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c1544b0 0x7f072c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:46.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.634+0000 7f072a7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c0c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:46.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.634+0000 7f072b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c1544b0 0x7f072c157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:60128/0 (socket says 172.21.15.129:60128) 2024-03-23T15:51:46.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.634+0000 7f072b7fe640 1 -- 172.21.15.129:0/2589962480 learned_addr learned my addr 172.21.15.129:0/2589962480 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:46.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.638+0000 7f072affd640 1 -- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 msgr2=0x7f072c0c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:46.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.638+0000 7f072affd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c0c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:46.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.638+0000 7f072affd640 1 -- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c1544b0 msgr2=0x7f072c157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:46.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.638+0000 7f072affd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c1544b0 0x7f072c157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:46.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.638+0000 7f072affd640 1 -- 172.21.15.129:0/2589962480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0720009400 con 0x7f072c06c0b0 2024-03-23T15:51:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.640+0000 7f072affd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c06c0b0 0x7f072c1f34d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f071c00e8a0 tx=0x7f071c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:46.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.644+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c009e60 con 0x7f072c06c0b0 2024-03-23T15:51:46.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.644+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f071c00c940 con 0x7f072c06c0b0 2024-03-23T15:51:46.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.644+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c00ccd0 con 0x7f072c06c0b0 2024-03-23T15:51:46.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.644+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c1f3c30 con 0x7f072c06c0b0 2024-03-23T15:51:46.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.644+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c1f41f0 con 0x7f072c06c0b0 2024-03-23T15:51:46.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.645+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f072c075740 con 0x7f072c06c0b0 2024-03-23T15:51:46.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.669+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f071c013470 con 0x7f072c06c0b0 2024-03-23T15:51:46.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.669+0000 7f07137fe640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f071806b810 0x7f071806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:46.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.669+0000 7f072a7fc640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f071806b810 0x7f071806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:46.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.670+0000 7f072a7fc640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f071806b810 0x7f071806dcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0720036000 tx=0x7f0720008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:46.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.672+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f071c014e40 con 0x7f072c06c0b0 2024-03-23T15:51:46.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:46.932+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f071c0553e0 con 0x7f072c06c0b0 2024-03-23T15:51:47.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.246+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f072c0b5b30 con 0x7f072c06c0b0 2024-03-23T15:51:47.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.266+0000 7f07137fe640 1 -- 172.21.15.129:0/2589962480 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f071c021090 con 0x7f072c06c0b0 2024-03-23T15:51:47.270 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:47.270 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:47.270 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f071806b810 msgr2=0x7f071806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f071806b810 0x7f071806dcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0720036000 tx=0x7f0720008030 comp rx=0 tx=0).stop 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c06c0b0 msgr2=0x7f072c1f34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c06c0b0 0x7f072c1f34d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f071c00e8a0 tx=0x7f071c00ed60 comp rx=0 tx=0).stop 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 shutdown_connections 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f071806b810 0x7f071806dcd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c1544b0 0x7f072c157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:47.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f072c0caa00 0x7f072c0c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:47.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 --2- 172.21.15.129:0/2589962480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c06c0b0 0x7f072c1f34d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:47.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 >> 172.21.15.129:0/2589962480 conn(0x7f072c061b50 msgr2=0x7f072c159440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:47.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 shutdown_connections 2024-03-23T15:51:47.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:47.269+0000 7f07317fd640 1 -- 172.21.15.129:0/2589962480 wait complete. 2024-03-23T15:51:47.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:47.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:48.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.989+0000 7fcc267e2640 1 Processor -- start 2024-03-23T15:51:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.989+0000 7fcc267e2640 1 -- start start 2024-03-23T15:51:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.990+0000 7fcc267e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ccd60 0x7fcc2015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.990+0000 7fcc267e2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc200cef40 0x7fcc2015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.990+0000 7fcc267e2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2015b290 con 0x7fcc200ceb00 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.990+0000 7fcc267e2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fcc200ce800 con 0x7fcc200cef40 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.990+0000 7fcc267e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc20154a70 con 0x7fcc200ccd60 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.991+0000 7fcc24d7f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc200cef40 0x7fcc2015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.991+0000 7fcc1ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ccd60 0x7fcc2015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.991+0000 7fcc24d7f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc200cef40 0x7fcc2015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53374/0 (socket says 172.21.15.129:53374) 2024-03-23T15:51:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.991+0000 7fcc24d7f640 1 -- 172.21.15.129:0/3349271899 learned_addr learned my addr 172.21.15.129:0/3349271899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:48.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.994+0000 7fcc1ffff640 1 -- 172.21.15.129:0/3349271899 >> v2:172.21.15.129:3300/0 conn(0x7fcc200cef40 msgr2=0x7fcc2015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:48.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.994+0000 7fcc1ffff640 1 --2- 172.21.15.129:0/3349271899 >> v2:172.21.15.129:3300/0 conn(0x7fcc200cef40 0x7fcc2015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:48.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.994+0000 7fcc1ffff640 1 -- 172.21.15.129:0/3349271899 >> v1:172.21.15.129:6789/0 conn(0x7fcc200ceb00 legacy=0x7fcc2006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:48.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.994+0000 7fcc1ffff640 1 -- 172.21.15.129:0/3349271899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc200cd950 con 0x7fcc200ccd60 2024-03-23T15:51:48.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.996+0000 7fcc1ffff640 1 --2- 172.21.15.129:0/3349271899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ccd60 0x7fcc2015ad50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc100097a0 tx=0x7fcc1002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a88885fe940ab121 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:49.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:48.998+0000 7fcc1effd640 1 -- 172.21.15.129:0/3349271899 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10030610 con 0x7fcc200ccd60 2024-03-23T15:51:49.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc1effd640 1 -- 172.21.15.129:0/3349271899 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10030770 con 0x7fcc200ccd60 2024-03-23T15:51:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc1effd640 1 -- 172.21.15.129:0/3349271899 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10030b00 con 0x7fcc200ccd60 2024-03-23T15:51:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 -- 172.21.15.129:0/3349271899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ccd60 msgr2=0x7fcc2015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 --2- 172.21.15.129:0/3349271899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ccd60 0x7fcc2015ad50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc100097a0 tx=0x7fcc1002b960 comp rx=0 tx=0).stop 2024-03-23T15:51:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 -- 172.21.15.129:0/3349271899 shutdown_connections 2024-03-23T15:51:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 --2- 172.21.15.129:0/3349271899 >> v2:172.21.15.129:3300/0 conn(0x7fcc200cef40 0x7fcc2015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 --2- 172.21.15.129:0/3349271899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ccd60 0x7fcc2015ad50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 -- 172.21.15.129:0/3349271899 >> 172.21.15.129:0/3349271899 conn(0x7fcc200b1840 msgr2=0x7fcc200b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 -- 172.21.15.129:0/3349271899 shutdown_connections 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.001+0000 7fcc267e2640 1 -- 172.21.15.129:0/3349271899 wait complete. 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 Processor -- start 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 -- start start 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc200ccd60 0x7fcc20157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ceb00 0x7fcc201f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200cef40 0x7fcc201e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2014bc10 con 0x7fcc200ccd60 2024-03-23T15:51:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200c9390 con 0x7fcc200cef40 2024-03-23T15:51:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.003+0000 7fcc267e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc200c7280 con 0x7fcc200ceb00 2024-03-23T15:51:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.004+0000 7fcc1f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ceb00 0x7fcc201f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.004+0000 7fcc1ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc200ccd60 0x7fcc20157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:49.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.004+0000 7fcc24d7f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200cef40 0x7fcc201e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:49.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.004+0000 7fcc1f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ceb00 0x7fcc201f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49542/0 (socket says 172.21.15.129:49542) 2024-03-23T15:51:49.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.004+0000 7fcc1f7fe640 1 -- 172.21.15.129:0/157686572 learned_addr learned my addr 172.21.15.129:0/157686572 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:49.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.008+0000 7fcc1ffff640 1 -- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ceb00 msgr2=0x7fcc201f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:49.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.008+0000 7fcc1ffff640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ceb00 0x7fcc201f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.008+0000 7fcc1ffff640 1 -- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200cef40 msgr2=0x7fcc201e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:49.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.008+0000 7fcc1ffff640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200cef40 0x7fcc201e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.008+0000 7fcc1ffff640 1 -- 172.21.15.129:0/157686572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc10009400 con 0x7fcc200ccd60 2024-03-23T15:51:49.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.010+0000 7fcc1ffff640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc200ccd60 0x7fcc20157190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc20154bb0 tx=0x7fcc10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:49.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.015+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1003e040 con 0x7fcc200ccd60 2024-03-23T15:51:49.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.015+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10039cb0 con 0x7fcc200ccd60 2024-03-23T15:51:49.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.015+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc200c7560 con 0x7fcc200ccd60 2024-03-23T15:51:49.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.015+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10043680 con 0x7fcc200ccd60 2024-03-23T15:51:49.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.016+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc201576d0 con 0x7fcc200ccd60 2024-03-23T15:51:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.017+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2004f530 con 0x7fcc200ccd60 2024-03-23T15:51:49.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.047+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc1004c3b0 con 0x7fcc200ccd60 2024-03-23T15:51:49.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.048+0000 7fcc1cff9640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b860 0x7fcc0c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:49.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.048+0000 7fcc1f7fe640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b860 0x7fcc0c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:49.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.049+0000 7fcc1f7fe640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b860 0x7fcc0c06dd20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcc20159b60 tx=0x7fcc14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:49.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.303+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcc10039e20 con 0x7fcc200ccd60 2024-03-23T15:51:49.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.303+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc100777d0 con 0x7fcc200ccd60 2024-03-23T15:51:49.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.532+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc200686f0 con 0x7fcc200ccd60 2024-03-23T15:51:49.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.545+0000 7fcc1cff9640 1 -- 172.21.15.129:0/157686572 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcc10034020 con 0x7fcc200ccd60 2024-03-23T15:51:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:49.549 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:49.549 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.547+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b860 msgr2=0x7fcc0c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.547+0000 7fcc267e2640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b860 0x7fcc0c06dd20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcc20159b60 tx=0x7fcc14009040 comp rx=0 tx=0).stop 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.547+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc200ccd60 msgr2=0x7fcc20157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.547+0000 7fcc267e2640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc200ccd60 0x7fcc20157190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc20154bb0 tx=0x7fcc10004730 comp rx=0 tx=0).stop 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 shutdown_connections 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b860 0x7fcc0c06dd20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200cef40 0x7fcc201e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200ceb00 0x7fcc201f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 --2- 172.21.15.129:0/157686572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc200ccd60 0x7fcc20157190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:49.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 >> 172.21.15.129:0/157686572 conn(0x7fcc200b1840 msgr2=0x7fcc20153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:49.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 shutdown_connections 2024-03-23T15:51:49.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:49.548+0000 7fcc267e2640 1 -- 172.21.15.129:0/157686572 wait complete. 2024-03-23T15:51:50.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:50.167 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:51.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:51.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.267+0000 7fe6c2581640 1 Processor -- start 2024-03-23T15:51:51.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.267+0000 7fe6c2581640 1 -- start start 2024-03-23T15:51:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.268+0000 7fe6c2581640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc15e040 0x7fe6bc06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.268+0000 7fe6c2581640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6bc06d2c0 con 0x7fe6bc06f570 2024-03-23T15:51:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.268+0000 7fe6c2581640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6bc15e740 con 0x7fe6bc15dc60 2024-03-23T15:51:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.268+0000 7fe6c2581640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc0ccd00 con 0x7fe6bc15e040 2024-03-23T15:51:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.268+0000 7fe6bbfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe6bc15dc60 0x7fe6bc0cdda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42728/0 (socket says 172.21.15.129:42728) 2024-03-23T15:51:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.268+0000 7fe6bbfff640 1 -- 172.21.15.129:0/1531048388 learned_addr learned my addr 172.21.15.129:0/1531048388 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.269+0000 7fe6bb7fe640 1 --2- 172.21.15.129:0/1531048388 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc15e040 0x7fe6bc06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.271+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458620165 0 0) 0x7fe6bc06d2c0 con 0x7fe6bc06f570 2024-03-23T15:51:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.271+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6a4003520 con 0x7fe6bc06f570 2024-03-23T15:51:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.272+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026622748 0 0) 0x7fe6bc15e740 con 0x7fe6bc15dc60 2024-03-23T15:51:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.272+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6bc06d2c0 con 0x7fe6bc15dc60 2024-03-23T15:51:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.273+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/1531048388 >> v1:172.21.15.157:6789/0 conn(0x7fe6bc15dc60 legacy=0x7fe6bc0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.273+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/1531048388 >> v1:172.21.15.129:6789/0 conn(0x7fe6bc06f570 legacy=0x7fe6bc0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.273+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/1531048388 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc0ccfe0 con 0x7fe6bc15e040 2024-03-23T15:51:51.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.276+0000 7fe6bb7fe640 1 --2- 172.21.15.129:0/1531048388 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc15e040 0x7fe6bc06cc50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00097d0 tx=0x7fe6b002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61bcc024d6e3e48c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:51.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.281+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0031800 con 0x7fe6bc15e040 2024-03-23T15:51:51.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.281+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b0031960 con 0x7fe6bc15e040 2024-03-23T15:51:51.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.281+0000 7fe6baffd640 1 -- 172.21.15.129:0/1531048388 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0031cf0 con 0x7fe6bc15e040 2024-03-23T15:51:51.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 -- 172.21.15.129:0/1531048388 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc15e040 msgr2=0x7fe6bc06cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 --2- 172.21.15.129:0/1531048388 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc15e040 0x7fe6bc06cc50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00097d0 tx=0x7fe6b002b8a0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 -- 172.21.15.129:0/1531048388 shutdown_connections 2024-03-23T15:51:51.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 --2- 172.21.15.129:0/1531048388 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc15e040 0x7fe6bc06cc50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 -- 172.21.15.129:0/1531048388 >> 172.21.15.129:0/1531048388 conn(0x7fe6bc071300 msgr2=0x7fe6bc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:51.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 -- 172.21.15.129:0/1531048388 shutdown_connections 2024-03-23T15:51:51.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.282+0000 7fe6c2581640 1 -- 172.21.15.129:0/1531048388 wait complete. 2024-03-23T15:51:51.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.283+0000 7fe6c2581640 1 Processor -- start 2024-03-23T15:51:51.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.283+0000 7fe6c2581640 1 -- start start 2024-03-23T15:51:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c2581640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc06f570 0x7fe6bc1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c2581640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 0x7fe6bc1f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c2581640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6bc15e040 0x7fe6bc070320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c2581640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc0c65c0 con 0x7fe6bc15e040 2024-03-23T15:51:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c2581640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc0c6910 con 0x7fe6bc15dc60 2024-03-23T15:51:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c2581640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe6bc1f3ce0 con 0x7fe6bc06f570 2024-03-23T15:51:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.284+0000 7fe6c0b1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6bc15e040 0x7fe6bc070320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.285+0000 7fe6bb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 0x7fe6bc1f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.285+0000 7fe6bbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc06f570 0x7fe6bc1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.285+0000 7fe6bb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 0x7fe6bc1f0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:53892/0 (socket says 172.21.15.129:53892) 2024-03-23T15:51:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.285+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/3632345333 learned_addr learned my addr 172.21.15.129:0/3632345333 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:51.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.289+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc06f570 msgr2=0x7fe6bc1f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.289+0000 7fe6bb7fe640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc06f570 0x7fe6bc1f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.289+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/3632345333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6bc15e040 msgr2=0x7fe6bc070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.289+0000 7fe6bb7fe640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6bc15e040 0x7fe6bc070320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.289+0000 7fe6bb7fe640 1 -- 172.21.15.129:0/3632345333 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b0009400 con 0x7fe6bc15dc60 2024-03-23T15:51:51.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.292+0000 7fe6bb7fe640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 0x7fe6bc1f0160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0000c00 tx=0x7fe6b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:51.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.294+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b003abe0 con 0x7fe6bc15dc60 2024-03-23T15:51:51.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.294+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc0c7b20 con 0x7fe6bc15dc60 2024-03-23T15:51:51.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.294+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc0c80c0 con 0x7fe6bc15dc60 2024-03-23T15:51:51.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.295+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6bc075740 con 0x7fe6bc15dc60 2024-03-23T15:51:51.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.297+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b003ad40 con 0x7fe6bc15dc60 2024-03-23T15:51:51.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.297+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b004d670 con 0x7fe6bc15dc60 2024-03-23T15:51:51.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.658+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6b004d7d0 con 0x7fe6bc15dc60 2024-03-23T15:51:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.658+0000 7fe6b8ff9640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe6a806b810 0x7fe6a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.659+0000 7fe6bbfff640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe6a806b810 0x7fe6a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.659+0000 7fe6bbfff640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe6a806b810 0x7fe6a806dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac005880 tx=0x7fe6ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:51.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.665+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe6b002e070 con 0x7fe6bc15dc60 2024-03-23T15:51:51.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.683+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6b0077170 con 0x7fe6bc15dc60 2024-03-23T15:51:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.906+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6bc0c90e0 con 0x7fe6bc15dc60 2024-03-23T15:51:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.927+0000 7fe6b8ff9640 1 -- 172.21.15.129:0/3632345333 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe6b0035020 con 0x7fe6bc15dc60 2024-03-23T15:51:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:51.931 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:51.931 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe6a806b810 msgr2=0x7fe6a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe6a806b810 0x7fe6a806dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac005880 tx=0x7fe6ac009040 comp rx=0 tx=0).stop 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 msgr2=0x7fe6bc1f0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 0x7fe6bc1f0160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0000c00 tx=0x7fe6b00046c0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 shutdown_connections 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6bc15e040 0x7fe6bc070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe6a806b810 0x7fe6a806dcd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6bc15dc60 0x7fe6bc1f0160 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 --2- 172.21.15.129:0/3632345333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6bc06f570 0x7fe6bc1f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:51.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 >> 172.21.15.129:0/3632345333 conn(0x7fe6bc071300 msgr2=0x7fe6bc152280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:51.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 shutdown_connections 2024-03-23T15:51:51.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:51.930+0000 7fe6c2581640 1 -- 172.21.15.129:0/3632345333 wait complete. 2024-03-23T15:51:52.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:52.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:53.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:53.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.649+0000 7fd257d7a640 1 Processor -- start 2024-03-23T15:51:53.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.649+0000 7fd257d7a640 1 -- start start 2024-03-23T15:51:53.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.651+0000 7fd257d7a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2501544b0 0x7fd25015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:53.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.651+0000 7fd257d7a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd2500caa00 0x7fd25006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:53.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.651+0000 7fd257d7a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2500cbcb0 con 0x7fd25006c0b0 2024-03-23T15:51:53.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.651+0000 7fd257d7a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd2500c5be0 con 0x7fd2500caa00 2024-03-23T15:51:53.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.651+0000 7fd257d7a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd25015b280 con 0x7fd2501544b0 2024-03-23T15:51:53.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.651+0000 7fd255315640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd2500caa00 0x7fd25006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:53.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.652+0000 7fd255b16640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2501544b0 0x7fd25015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:53.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.652+0000 7fd256317640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd25006c0b0 0x7fd2500c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44530/0 (socket says 172.21.15.129:44530) 2024-03-23T15:51:53.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.652+0000 7fd256317640 1 -- 172.21.15.129:0/4245063516 learned_addr learned my addr 172.21.15.129:0/4245063516 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:53.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.654+0000 7fd255b16640 1 -- 172.21.15.129:0/4245063516 >> v2:172.21.15.129:3300/0 conn(0x7fd2500caa00 msgr2=0x7fd25006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:53.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.654+0000 7fd255b16640 1 --2- 172.21.15.129:0/4245063516 >> v2:172.21.15.129:3300/0 conn(0x7fd2500caa00 0x7fd25006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:53.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.654+0000 7fd255b16640 1 -- 172.21.15.129:0/4245063516 >> v1:172.21.15.129:6789/0 conn(0x7fd25006c0b0 legacy=0x7fd2500c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:53.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.654+0000 7fd255b16640 1 -- 172.21.15.129:0/4245063516 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25015a9e0 con 0x7fd2501544b0 2024-03-23T15:51:53.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.657+0000 7fd255b16640 1 --2- 172.21.15.129:0/4245063516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2501544b0 0x7fd25015a2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd25006cd70 tx=0x7fd24c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f8ae9ffd42259d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.661+0000 7fd254b14640 1 -- 172.21.15.129:0/4245063516 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c030610 con 0x7fd2501544b0 2024-03-23T15:51:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.661+0000 7fd254b14640 1 -- 172.21.15.129:0/4245063516 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24c030770 con 0x7fd2501544b0 2024-03-23T15:51:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.661+0000 7fd254b14640 1 -- 172.21.15.129:0/4245063516 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c030b00 con 0x7fd2501544b0 2024-03-23T15:51:53.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 -- 172.21.15.129:0/4245063516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2501544b0 msgr2=0x7fd25015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:53.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 --2- 172.21.15.129:0/4245063516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2501544b0 0x7fd25015a2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd25006cd70 tx=0x7fd24c02b960 comp rx=0 tx=0).stop 2024-03-23T15:51:53.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 -- 172.21.15.129:0/4245063516 shutdown_connections 2024-03-23T15:51:53.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 --2- 172.21.15.129:0/4245063516 >> v2:172.21.15.129:3300/0 conn(0x7fd2500caa00 0x7fd25006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:53.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 --2- 172.21.15.129:0/4245063516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2501544b0 0x7fd25015a2e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:53.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 -- 172.21.15.129:0/4245063516 >> 172.21.15.129:0/4245063516 conn(0x7fd250061b50 msgr2=0x7fd250061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:53.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 -- 172.21.15.129:0/4245063516 shutdown_connections 2024-03-23T15:51:53.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.662+0000 7fd257d7a640 1 -- 172.21.15.129:0/4245063516 wait complete. 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.663+0000 7fd257d7a640 1 Processor -- start 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.663+0000 7fd257d7a640 1 -- start start 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.664+0000 7fd257d7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd25006c0b0 0x7fd250155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.664+0000 7fd257d7a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2500caa00 0x7fd250154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.664+0000 7fd257d7a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2501544b0 0x7fd2501476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.664+0000 7fd257d7a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501561c0 con 0x7fd25006c0b0 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.664+0000 7fd257d7a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd250157430 con 0x7fd2500caa00 2024-03-23T15:51:53.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.664+0000 7fd257d7a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2501576f0 con 0x7fd2501544b0 2024-03-23T15:51:53.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.665+0000 7fd255b16640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd25006c0b0 0x7fd250155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:53.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.665+0000 7fd256317640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2501544b0 0x7fd2501476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:53.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.665+0000 7fd255b16640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd25006c0b0 0x7fd250155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47248/0 (socket says 172.21.15.129:47248) 2024-03-23T15:51:53.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.665+0000 7fd255b16640 1 -- 172.21.15.129:0/3857679316 learned_addr learned my addr 172.21.15.129:0/3857679316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:53.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.665+0000 7fd255315640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2500caa00 0x7fd250154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:53.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.669+0000 7fd255315640 1 -- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2501544b0 msgr2=0x7fd2501476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:53.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.669+0000 7fd255315640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2501544b0 0x7fd2501476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:53.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.669+0000 7fd255315640 1 -- 172.21.15.129:0/3857679316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd25006c0b0 msgr2=0x7fd250155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:53.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.669+0000 7fd255315640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd25006c0b0 0x7fd250155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:53.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.669+0000 7fd255315640 1 -- 172.21.15.129:0/3857679316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c009400 con 0x7fd2500caa00 2024-03-23T15:51:53.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.672+0000 7fd255315640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2500caa00 0x7fd250154f60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd24000e850 tx=0x7fd24000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:53.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.674+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2400095b0 con 0x7fd2500caa00 2024-03-23T15:51:53.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.674+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2501579b0 con 0x7fd2500caa00 2024-03-23T15:51:53.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.674+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd250156900 con 0x7fd2500caa00 2024-03-23T15:51:53.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.675+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd250075740 con 0x7fd2500caa00 2024-03-23T15:51:53.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.950+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd240005990 con 0x7fd2500caa00 2024-03-23T15:51:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.950+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240005d00 con 0x7fd2500caa00 2024-03-23T15:51:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.951+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd24000c940 con 0x7fd2500caa00 2024-03-23T15:51:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.951+0000 7fd2467fc640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd23406b840 0x7fd23406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.952+0000 7fd255b16640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd23406b840 0x7fd23406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.952+0000 7fd255b16640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd23406b840 0x7fd23406dd00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd24c02b9a0 tx=0x7fd24c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:53.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.955+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd2400592e0 con 0x7fd2500caa00 2024-03-23T15:51:53.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:53.965+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd240055580 con 0x7fd2500caa00 2024-03-23T15:51:54.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.183+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd250156400 con 0x7fd2500caa00 2024-03-23T15:51:54.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.207+0000 7fd2467fc640 1 -- 172.21.15.129:0/3857679316 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd240021020 con 0x7fd2500caa00 2024-03-23T15:51:54.211 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:54.211 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:54.211 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.209+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd23406b840 msgr2=0x7fd23406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.209+0000 7fd257d7a640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd23406b840 0x7fd23406dd00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd24c02b9a0 tx=0x7fd24c03b040 comp rx=0 tx=0).stop 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.209+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2500caa00 msgr2=0x7fd250154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.209+0000 7fd257d7a640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2500caa00 0x7fd250154f60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd24000e850 tx=0x7fd24000ed10 comp rx=0 tx=0).stop 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 shutdown_connections 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2501544b0 0x7fd2501476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd23406b840 0x7fd23406dd00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2500caa00 0x7fd250154f60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 --2- 172.21.15.129:0/3857679316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd25006c0b0 0x7fd250155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:54.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 >> 172.21.15.129:0/3857679316 conn(0x7fd250061b50 msgr2=0x7fd250153650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 shutdown_connections 2024-03-23T15:51:54.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:54.210+0000 7fd257d7a640 1 -- 172.21.15.129:0/3857679316 wait complete. 2024-03-23T15:51:54.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:54.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:55.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:55.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.929+0000 7f0b45aed640 1 Processor -- start 2024-03-23T15:51:55.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.929+0000 7f0b45aed640 1 -- start start 2024-03-23T15:51:55.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.929+0000 7f0b45aed640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0b400ccd60 0x7f0b4015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:55.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.929+0000 7f0b45aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400cef40 0x7f0b4015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:55.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.929+0000 7f0b45aed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4015b290 con 0x7f0b400ceb00 2024-03-23T15:51:55.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.929+0000 7f0b45aed640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f0b400ce800 con 0x7f0b400ccd60 2024-03-23T15:51:55.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.930+0000 7f0b45aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b40154a70 con 0x7f0b400cef40 2024-03-23T15:51:55.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.930+0000 7f0b3f7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0b400ccd60 0x7f0b4015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:55.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.930+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400cef40 0x7f0b4015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:55.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.930+0000 7f0b3effd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f0b400ceb00 0x7f0b4006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44534/0 (socket says 172.21.15.129:44534) 2024-03-23T15:51:55.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.930+0000 7f0b3effd640 1 -- 172.21.15.129:0/2427699098 learned_addr learned my addr 172.21.15.129:0/2427699098 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:51:55.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.933+0000 7f0b3ffff640 1 -- 172.21.15.129:0/2427699098 >> v2:172.21.15.129:3300/0 conn(0x7f0b400ccd60 msgr2=0x7f0b4015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:55.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.933+0000 7f0b3ffff640 1 --2- 172.21.15.129:0/2427699098 >> v2:172.21.15.129:3300/0 conn(0x7f0b400ccd60 0x7f0b4015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:55.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.933+0000 7f0b3ffff640 1 -- 172.21.15.129:0/2427699098 >> v1:172.21.15.129:6789/0 conn(0x7f0b400ceb00 legacy=0x7f0b4006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:55.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.933+0000 7f0b3ffff640 1 -- 172.21.15.129:0/2427699098 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b400cd950 con 0x7f0b400cef40 2024-03-23T15:51:55.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.935+0000 7f0b3ffff640 1 --2- 172.21.15.129:0/2427699098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400cef40 0x7f0b4015b8f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009d90 tx=0x7f0b3802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6795b0238d0072 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:55.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.939+0000 7f0b3e7fc640 1 -- 172.21.15.129:0/2427699098 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3802cce0 con 0x7f0b400cef40 2024-03-23T15:51:55.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.939+0000 7f0b3e7fc640 1 -- 172.21.15.129:0/2427699098 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3802ce40 con 0x7f0b400cef40 2024-03-23T15:51:55.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.939+0000 7f0b3e7fc640 1 -- 172.21.15.129:0/2427699098 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38034660 con 0x7f0b400cef40 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 -- 172.21.15.129:0/2427699098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400cef40 msgr2=0x7f0b4015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2427699098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400cef40 0x7f0b4015b8f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009d90 tx=0x7f0b3802bf30 comp rx=0 tx=0).stop 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 -- 172.21.15.129:0/2427699098 shutdown_connections 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2427699098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400cef40 0x7f0b4015b8f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2427699098 >> v2:172.21.15.129:3300/0 conn(0x7f0b400ccd60 0x7f0b4015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 -- 172.21.15.129:0/2427699098 >> 172.21.15.129:0/2427699098 conn(0x7f0b400b1840 msgr2=0x7f0b400b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 -- 172.21.15.129:0/2427699098 shutdown_connections 2024-03-23T15:51:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.940+0000 7f0b45aed640 1 -- 172.21.15.129:0/2427699098 wait complete. 2024-03-23T15:51:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.942+0000 7f0b45aed640 1 Processor -- start 2024-03-23T15:51:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.943+0000 7f0b45aed640 1 -- start start 2024-03-23T15:51:55.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b45aed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b400ccd60 0x7f0b40150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:55.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b45aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400ceb00 0x7f0b4015bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b45aed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b400cef40 0x7f0b4015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b45aed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0b400c93b0 con 0x7f0b400ccd60 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b45aed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401f4300 con 0x7f0b400cef40 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b45aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b401f3d60 con 0x7f0b400ceb00 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400ceb00 0x7f0b4015bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.944+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b400cef40 0x7f0b4015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.945+0000 7f0b3effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400ceb00 0x7f0b4015bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:33562/0 (socket says 172.21.15.129:33562) 2024-03-23T15:51:55.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.945+0000 7f0b3effd640 1 -- 172.21.15.129:0/2248837127 learned_addr learned my addr 172.21.15.129:0/2248837127 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:55.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.946+0000 7f0b3f7fe640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b400ccd60 0x7f0b40150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:55.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.949+0000 7f0b3ffff640 1 -- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400ceb00 msgr2=0x7f0b4015bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:55.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.949+0000 7f0b3ffff640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400ceb00 0x7f0b4015bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:55.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.949+0000 7f0b3ffff640 1 -- 172.21.15.129:0/2248837127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b400ccd60 msgr2=0x7f0b40150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:55.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.949+0000 7f0b3ffff640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b400ccd60 0x7f0b40150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:55.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.949+0000 7f0b3ffff640 1 -- 172.21.15.129:0/2248837127 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b38009a70 con 0x7f0b400cef40 2024-03-23T15:51:55.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.951+0000 7f0b3ffff640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b400cef40 0x7f0b4015c200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0b4015e820 tx=0x7f0b38004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.954+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3803d600 con 0x7f0b400cef40 2024-03-23T15:51:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.954+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b400c6dd0 con 0x7f0b400cef40 2024-03-23T15:51:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.954+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4015caf0 con 0x7f0b400cef40 2024-03-23T15:51:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.954+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4004f530 con 0x7f0b400cef40 2024-03-23T15:51:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.956+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b38009450 con 0x7f0b400cef40 2024-03-23T15:51:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.956+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3803d980 con 0x7f0b400cef40 2024-03-23T15:51:55.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.979+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0b3803b040 con 0x7f0b400cef40 2024-03-23T15:51:55.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.980+0000 7f0b27fff640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0b2c06b810 0x7f0b2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:55.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.980+0000 7f0b3f7fe640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0b2c06b810 0x7f0b2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:55.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:55.980+0000 7f0b3f7fe640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0b2c06b810 0x7f0b2c06dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009b40 tx=0x7f0b30009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:56.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.226+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0b3807afd0 con 0x7f0b400cef40 2024-03-23T15:51:56.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.244+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0b38077270 con 0x7f0b400cef40 2024-03-23T15:51:56.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.460+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b4006db50 con 0x7f0b400cef40 2024-03-23T15:51:56.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.486+0000 7f0b27fff640 1 -- 172.21.15.129:0/2248837127 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0b38039090 con 0x7f0b400cef40 2024-03-23T15:51:56.489 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:56.489 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:56.489 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0b2c06b810 msgr2=0x7f0b2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0b2c06b810 0x7f0b2c06dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009b40 tx=0x7f0b30009190 comp rx=0 tx=0).stop 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b400cef40 msgr2=0x7f0b4015c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b400cef40 0x7f0b4015c200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0b4015e820 tx=0x7f0b38004a40 comp rx=0 tx=0).stop 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 shutdown_connections 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0b2c06b810 0x7f0b2c06dcd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b400cef40 0x7f0b4015c200 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:56.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b400ceb00 0x7f0b4015bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:56.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 --2- 172.21.15.129:0/2248837127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b400ccd60 0x7f0b40150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:56.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 >> 172.21.15.129:0/2248837127 conn(0x7f0b400b1840 msgr2=0x7f0b400b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:56.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 shutdown_connections 2024-03-23T15:51:56.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:56.488+0000 7f0b45aed640 1 -- 172.21.15.129:0/2248837127 wait complete. 2024-03-23T15:51:57.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:57.107 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:51:58.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:51:58.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.207+0000 7f06fb44d640 1 Processor -- start 2024-03-23T15:51:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.207+0000 7f06fb44d640 1 -- start start 2024-03-23T15:51:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.208+0000 7f06fb44d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f415e520 0x7f06f415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.208+0000 7f06fb44d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f06f406bd40 0x7f06f40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.208+0000 7f06fb44d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f415e090 con 0x7f06f4154340 2024-03-23T15:51:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.208+0000 7f06fb44d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f06f415d860 con 0x7f06f406bd40 2024-03-23T15:51:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.208+0000 7f06fb44d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06f406ca10 con 0x7f06f415e520 2024-03-23T15:51:58.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.209+0000 7f06f89e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f415e520 0x7f06f415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:58.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.209+0000 7f06f99ea640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f06f406bd40 0x7f06f40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:58.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.209+0000 7f06f99ea640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f06f406bd40 0x7f06f40cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:53942/0 (socket says 172.21.15.129:53942) 2024-03-23T15:51:58.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.209+0000 7f06f99ea640 1 -- 172.21.15.129:0/4195216877 learned_addr learned my addr 172.21.15.129:0/4195216877 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:58.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.212+0000 7f06f99ea640 1 -- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f415e520 msgr2=0x7f06f415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:51:58.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.212+0000 7f06f99ea640 1 --2- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f415e520 0x7f06f415aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.212+0000 7f06f99ea640 1 -- 172.21.15.129:0/4195216877 >> v1:172.21.15.157:6789/0 conn(0x7f06f4154340 legacy=0x7f06f40c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:58.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.212+0000 7f06f99ea640 1 -- 172.21.15.129:0/4195216877 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f4154720 con 0x7f06f406bd40 2024-03-23T15:51:58.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.219+0000 7f06f99ea640 1 --2- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f406bd40 0x7f06f40cc160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009e50 tx=0x7f06ec02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3bc092aad0e2b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:58.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.225+0000 7f06ebfff640 1 -- 172.21.15.129:0/4195216877 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec02cb90 con 0x7f06f406bd40 2024-03-23T15:51:58.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.225+0000 7f06ebfff640 1 -- 172.21.15.129:0/4195216877 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ec02ccf0 con 0x7f06f406bd40 2024-03-23T15:51:58.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.225+0000 7f06ebfff640 1 -- 172.21.15.129:0/4195216877 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec034610 con 0x7f06f406bd40 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 -- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f406bd40 msgr2=0x7f06f40cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 --2- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f406bd40 0x7f06f40cc160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009e50 tx=0x7f06ec02bff0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 -- 172.21.15.129:0/4195216877 shutdown_connections 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 --2- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f406bd40 0x7f06f40cc160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 --2- 172.21.15.129:0/4195216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f415e520 0x7f06f415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 -- 172.21.15.129:0/4195216877 >> 172.21.15.129:0/4195216877 conn(0x7f06f4071dc0 msgr2=0x7f06f40721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 -- 172.21.15.129:0/4195216877 shutdown_connections 2024-03-23T15:51:58.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.226+0000 7f06fb44d640 1 -- 172.21.15.129:0/4195216877 wait complete. 2024-03-23T15:51:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.228+0000 7f06fb44d640 1 Processor -- start 2024-03-23T15:51:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.228+0000 7f06fb44d640 1 -- start start 2024-03-23T15:51:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.229+0000 7f06fb44d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 0x7f06f41e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.229+0000 7f06fb44d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f4154340 0x7f06f41f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.229+0000 7f06fb44d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f415e520 0x7f06f4150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.229+0000 7f06fb44d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41e5100 con 0x7f06f406bd40 2024-03-23T15:51:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.229+0000 7f06fb44d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41f0950 con 0x7f06f415e520 2024-03-23T15:51:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.229+0000 7f06fb44d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06f41e36c0 con 0x7f06f4154340 2024-03-23T15:51:58.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.230+0000 7f06f91e9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 0x7f06f41e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:58.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.230+0000 7f06f91e9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 0x7f06f41e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47284/0 (socket says 172.21.15.129:47284) 2024-03-23T15:51:58.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.230+0000 7f06f91e9640 1 -- 172.21.15.129:0/1528938648 learned_addr learned my addr 172.21.15.129:0/1528938648 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:51:58.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.230+0000 7f06f89e8640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f4154340 0x7f06f41f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:58.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.231+0000 7f06f99ea640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f415e520 0x7f06f4150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.235+0000 7f06f91e9640 1 -- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f4154340 msgr2=0x7f06f41f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.235+0000 7f06f91e9640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f4154340 0x7f06f41f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.235+0000 7f06f91e9640 1 -- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f415e520 msgr2=0x7f06f4150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.235+0000 7f06f91e9640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f415e520 0x7f06f4150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.235+0000 7f06f91e9640 1 -- 172.21.15.129:0/1528938648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ec009b30 con 0x7f06f406bd40 2024-03-23T15:51:58.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.239+0000 7f06f91e9640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 0x7f06f41e4bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c400 tx=0x7f06f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:58.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.243+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f00052d0 con 0x7f06f406bd40 2024-03-23T15:51:58.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.243+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f0005450 con 0x7f06f406bd40 2024-03-23T15:51:58.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.243+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0007260 con 0x7f06f406bd40 2024-03-23T15:51:58.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.243+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f41e3910 con 0x7f06f406bd40 2024-03-23T15:51:58.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.243+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f40ca200 con 0x7f06f406bd40 2024-03-23T15:51:58.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.244+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f4075740 con 0x7f06f406bd40 2024-03-23T15:51:58.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.508+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f06f00073c0 con 0x7f06f406bd40 2024-03-23T15:51:58.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.509+0000 7f06e9ffb640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f06d806b810 0x7f06d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:51:58.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.509+0000 7f06f89e8640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f06d806b810 0x7f06d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:51:58.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.510+0000 7f06f89e8640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f06d806b810 0x7f06d806dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f06f4151130 tx=0x7f06e4009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:51:58.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.530+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f06f0007670 con 0x7f06f406bd40 2024-03-23T15:51:58.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.530+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f06f00548c0 con 0x7f06f406bd40 2024-03-23T15:51:58.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.751+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06f406bb20 con 0x7f06f406bd40 2024-03-23T15:51:58.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.770+0000 7f06e9ffb640 1 -- 172.21.15.129:0/1528938648 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f06f0020020 con 0x7f06f406bd40 2024-03-23T15:51:58.773 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:51:58.773 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:51:58.774 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f06d806b810 msgr2=0x7f06d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f06d806b810 0x7f06d806dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f06f4151130 tx=0x7f06e4009110 comp rx=0 tx=0).stop 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 msgr2=0x7f06f41e4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 0x7f06f41e4bc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c400 tx=0x7f06f000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 shutdown_connections 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f06d806b810 0x7f06d806dcd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06f415e520 0x7f06f4150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06f4154340 0x7f06f41f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 --2- 172.21.15.129:0/1528938648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06f406bd40 0x7f06f41e4bc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:51:58.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 >> 172.21.15.129:0/1528938648 conn(0x7f06f4071dc0 msgr2=0x7f06f4152830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:51:58.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 shutdown_connections 2024-03-23T15:51:58.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:51:58.772+0000 7f06fb44d640 1 -- 172.21.15.129:0/1528938648 wait complete. 2024-03-23T15:51:59.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:51:59.392 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:00.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:00.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.493+0000 7f72e72c9640 1 Processor -- start 2024-03-23T15:52:00.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.494+0000 7f72e72c9640 1 -- start start 2024-03-23T15:52:00.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.494+0000 7f72e72c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f72e00c7380 0x7f72e00c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:00.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.494+0000 7f72e72c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f72e00c8e90 0x7f72e00c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:00.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.494+0000 7f72e72c9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e00c3bd0 con 0x7f72e0156f60 2024-03-23T15:52:00.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.494+0000 7f72e72c9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f72e00c7ca0 con 0x7f72e00c7380 2024-03-23T15:52:00.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.494+0000 7f72e72c9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f72e00c3740 con 0x7f72e00c8e90 2024-03-23T15:52:00.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.495+0000 7f72e4864640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f72e00c8e90 0x7f72e00c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:00.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.496+0000 7f72e4864640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f72e00c8e90 0x7f72e00c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:34984/0 (socket says 172.21.15.129:34984) 2024-03-23T15:52:00.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.496+0000 7f72e4864640 1 -- 172.21.15.129:0/4127629442 learned_addr learned my addr 172.21.15.129:0/4127629442 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:00.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.497+0000 7f72e5065640 1 --2- 172.21.15.129:0/4127629442 >> v2:172.21.15.129:3300/0 conn(0x7f72e00c7380 0x7f72e00c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:00.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.498+0000 7f72e4864640 1 -- 172.21.15.129:0/4127629442 >> v2:172.21.15.129:3300/0 conn(0x7f72e00c7380 msgr2=0x7f72e00c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:00.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.498+0000 7f72e4864640 1 --2- 172.21.15.129:0/4127629442 >> v2:172.21.15.129:3300/0 conn(0x7f72e00c7380 0x7f72e00c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:00.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.498+0000 7f72e4864640 1 -- 172.21.15.129:0/4127629442 >> v1:172.21.15.129:6789/0 conn(0x7f72e0156f60 legacy=0x7f72e0157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:00.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.498+0000 7f72e4864640 1 -- 172.21.15.129:0/4127629442 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e00c80c0 con 0x7f72e00c8e90 2024-03-23T15:52:00.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.503+0000 7f72e4864640 1 --2- 172.21.15.129:0/4127629442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c8e90 0x7f72e00c50c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f72e0062c00 tx=0x7f72d002e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce74c754efc581d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:00.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.505+0000 7f72d7fff640 1 -- 172.21.15.129:0/4127629442 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d002ee90 con 0x7f72e00c8e90 2024-03-23T15:52:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.507+0000 7f72d7fff640 1 -- 172.21.15.129:0/4127629442 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d0036440 con 0x7f72e00c8e90 2024-03-23T15:52:00.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.507+0000 7f72d7fff640 1 -- 172.21.15.129:0/4127629442 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d00367d0 con 0x7f72e00c8e90 2024-03-23T15:52:00.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 -- 172.21.15.129:0/4127629442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c8e90 msgr2=0x7f72e00c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:00.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 --2- 172.21.15.129:0/4127629442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c8e90 0x7f72e00c50c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f72e0062c00 tx=0x7f72d002e0e0 comp rx=0 tx=0).stop 2024-03-23T15:52:00.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 -- 172.21.15.129:0/4127629442 shutdown_connections 2024-03-23T15:52:00.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 --2- 172.21.15.129:0/4127629442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c8e90 0x7f72e00c50c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:00.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 --2- 172.21.15.129:0/4127629442 >> v2:172.21.15.129:3300/0 conn(0x7f72e00c7380 0x7f72e00c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:00.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 -- 172.21.15.129:0/4127629442 >> 172.21.15.129:0/4127629442 conn(0x7f72e00c8370 msgr2=0x7f72e00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:00.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 -- 172.21.15.129:0/4127629442 shutdown_connections 2024-03-23T15:52:00.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.508+0000 7f72e72c9640 1 -- 172.21.15.129:0/4127629442 wait complete. 2024-03-23T15:52:00.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 Processor -- start 2024-03-23T15:52:00.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 -- start start 2024-03-23T15:52:00.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c7380 0x7f72e00caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:00.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 0x7f72e01f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:00.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72e0156f60 0x7f72e01f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:00.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72e013fff0 con 0x7f72e00c8e90 2024-03-23T15:52:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0140520 con 0x7f72e00c7380 2024-03-23T15:52:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.510+0000 7f72e72c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f72e0155880 con 0x7f72e0156f60 2024-03-23T15:52:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.511+0000 7f72e4864640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 0x7f72e01f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.511+0000 7f72e5065640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c7380 0x7f72e00caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.511+0000 7f72e4864640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 0x7f72e01f3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42024/0 (socket says 172.21.15.129:42024) 2024-03-23T15:52:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.511+0000 7f72e4864640 1 -- 172.21.15.129:0/289648213 learned_addr learned my addr 172.21.15.129:0/289648213 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.511+0000 7f72e5866640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72e0156f60 0x7f72e01f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:00.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.515+0000 7f72e4864640 1 -- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72e0156f60 msgr2=0x7f72e01f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:00.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.515+0000 7f72e4864640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72e0156f60 0x7f72e01f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:00.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.515+0000 7f72e4864640 1 -- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c7380 msgr2=0x7f72e00caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:00.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.515+0000 7f72e4864640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c7380 0x7f72e00caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:00.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.515+0000 7f72e4864640 1 -- 172.21.15.129:0/289648213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d000b770 con 0x7f72e00c8e90 2024-03-23T15:52:00.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.515+0000 7f72e5065640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c7380 0x7f72e00caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:00.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.518+0000 7f72e4864640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 0x7f72e01f3680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f72d0000c00 tx=0x7f72d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:00.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.522+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0034850 con 0x7f72e00c8e90 2024-03-23T15:52:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.522+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d00349d0 con 0x7f72e00c8e90 2024-03-23T15:52:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.522+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e0151240 con 0x7f72e00c8e90 2024-03-23T15:52:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.522+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0034d80 con 0x7f72e00c8e90 2024-03-23T15:52:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.522+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e0155510 con 0x7f72e00c8e90 2024-03-23T15:52:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.523+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e01559c0 con 0x7f72e00c8e90 2024-03-23T15:52:00.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.543+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f72d00403e0 con 0x7f72e00c8e90 2024-03-23T15:52:00.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.543+0000 7f72d5ffb640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f72c406b860 0x7f72c406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:00.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.544+0000 7f72e5065640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f72c406b860 0x7f72c406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:00.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.544+0000 7f72e5065640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f72c406b860 0x7f72c406dd20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f72dc006fd0 tx=0x7f72dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.803+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f72d007b5c0 con 0x7f72e00c8e90 2024-03-23T15:52:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:00.803+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f72d0077860 con 0x7f72e00c8e90 2024-03-23T15:52:01.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.037+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72e01556a0 con 0x7f72e00c8e90 2024-03-23T15:52:01.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.058+0000 7f72d5ffb640 1 -- 172.21.15.129:0/289648213 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f72d0046020 con 0x7f72e00c8e90 2024-03-23T15:52:01.061 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:01.061 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:01.061 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:01.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.060+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f72c406b860 msgr2=0x7f72c406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:01.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.060+0000 7f72e72c9640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f72c406b860 0x7f72c406dd20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f72dc006fd0 tx=0x7f72dc008040 comp rx=0 tx=0).stop 2024-03-23T15:52:01.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.060+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 msgr2=0x7f72e01f3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:01.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.060+0000 7f72e72c9640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 0x7f72e01f3680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f72d0000c00 tx=0x7f72d00046c0 comp rx=0 tx=0).stop 2024-03-23T15:52:01.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 shutdown_connections 2024-03-23T15:52:01.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72e0156f60 0x7f72e01f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:01.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72e00c8e90 0x7f72e01f3680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:01.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f72c406b860 0x7f72c406dd20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:01.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 --2- 172.21.15.129:0/289648213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72e00c7380 0x7f72e00caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:01.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 >> 172.21.15.129:0/289648213 conn(0x7f72e00c8370 msgr2=0x7f72e0072370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:01.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 shutdown_connections 2024-03-23T15:52:01.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:01.061+0000 7f72e72c9640 1 -- 172.21.15.129:0/289648213 wait complete. 2024-03-23T15:52:01.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:01.682 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:02.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:02.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.783+0000 7fb72db3a640 1 Processor -- start 2024-03-23T15:52:02.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.783+0000 7fb72db3a640 1 -- start start 2024-03-23T15:52:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.784+0000 7fb72db3a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb728154270 0x7fb7280ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.784+0000 7fb72db3a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb72815d880 con 0x7fb72815ad10 2024-03-23T15:52:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.784+0000 7fb72db3a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7280c4ae0 con 0x7fb7280c4f70 2024-03-23T15:52:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.784+0000 7fb72db3a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb7280cd5e0 con 0x7fb728154270 2024-03-23T15:52:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.785+0000 7fb726ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb72815ad10 0x7fb72806c110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:35066/0 (socket says 172.21.15.129:35066) 2024-03-23T15:52:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.785+0000 7fb726ffd640 1 -- 172.21.15.129:0/3203921024 learned_addr learned my addr 172.21.15.129:0/3203921024 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.785+0000 7fb7277fe640 1 --2- 172.21.15.129:0/3203921024 >> v2:172.21.15.157:3300/0 conn(0x7fb728154270 0x7fb7280ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.787+0000 7fb7267fc640 1 -- 172.21.15.129:0/3203921024 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680856087 0 0) 0x7fb72815d880 con 0x7fb72815ad10 2024-03-23T15:52:02.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.787+0000 7fb7267fc640 1 -- 172.21.15.129:0/3203921024 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb710003520 con 0x7fb72815ad10 2024-03-23T15:52:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.788+0000 7fb7277fe640 1 -- 172.21.15.129:0/3203921024 >> v1:172.21.15.157:6789/0 conn(0x7fb7280c4f70 legacy=0x7fb7280c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.788+0000 7fb7277fe640 1 -- 172.21.15.129:0/3203921024 >> v1:172.21.15.129:6789/0 conn(0x7fb72815ad10 legacy=0x7fb72806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.788+0000 7fb7277fe640 1 -- 172.21.15.129:0/3203921024 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72815dab0 con 0x7fb728154270 2024-03-23T15:52:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.793+0000 7fb7277fe640 1 --2- 172.21.15.129:0/3203921024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb728154270 0x7fb7280ca580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb7180097a0 tx=0x7fb71802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=310164d86102a5a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.797+0000 7fb7267fc640 1 -- 172.21.15.129:0/3203921024 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71802e070 con 0x7fb728154270 2024-03-23T15:52:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.797+0000 7fb7267fc640 1 -- 172.21.15.129:0/3203921024 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7180315a0 con 0x7fb728154270 2024-03-23T15:52:02.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.797+0000 7fb7267fc640 1 -- 172.21.15.129:0/3203921024 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718031930 con 0x7fb728154270 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 -- 172.21.15.129:0/3203921024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb728154270 msgr2=0x7fb7280ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 --2- 172.21.15.129:0/3203921024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb728154270 0x7fb7280ca580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb7180097a0 tx=0x7fb71802b840 comp rx=0 tx=0).stop 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 -- 172.21.15.129:0/3203921024 shutdown_connections 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 --2- 172.21.15.129:0/3203921024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb728154270 0x7fb7280ca580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 -- 172.21.15.129:0/3203921024 >> 172.21.15.129:0/3203921024 conn(0x7fb72806d290 msgr2=0x7fb728072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 -- 172.21.15.129:0/3203921024 shutdown_connections 2024-03-23T15:52:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.798+0000 7fb72db3a640 1 -- 172.21.15.129:0/3203921024 wait complete. 2024-03-23T15:52:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.800+0000 7fb72db3a640 1 Processor -- start 2024-03-23T15:52:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.800+0000 7fb72db3a640 1 -- start start 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.801+0000 7fb72db3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 0x7fb72815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.801+0000 7fb72db3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb728154270 0x7fb72814ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.801+0000 7fb72db3a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb72815ad10 0x7fb7281f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.801+0000 7fb72db3a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281f4570 con 0x7fb7280c4f70 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.801+0000 7fb72db3a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb72815b4f0 con 0x7fb72815ad10 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.801+0000 7fb72db3a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb72815b790 con 0x7fb728154270 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.802+0000 7fb7277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 0x7fb72815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.802+0000 7fb726ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb728154270 0x7fb72814ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.802+0000 7fb727fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb72815ad10 0x7fb7281f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.802+0000 7fb7277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 0x7fb72815bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42034/0 (socket says 172.21.15.129:42034) 2024-03-23T15:52:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.802+0000 7fb7277fe640 1 -- 172.21.15.129:0/2872739411 learned_addr learned my addr 172.21.15.129:0/2872739411 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.806+0000 7fb7277fe640 1 -- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb728154270 msgr2=0x7fb72814ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.806+0000 7fb7277fe640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb728154270 0x7fb72814ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.806+0000 7fb7277fe640 1 -- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb72815ad10 msgr2=0x7fb7281f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.806+0000 7fb7277fe640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb72815ad10 0x7fb7281f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.806+0000 7fb7277fe640 1 -- 172.21.15.129:0/2872739411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb718009400 con 0x7fb7280c4f70 2024-03-23T15:52:02.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.808+0000 7fb7277fe640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 0x7fb72815bd00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb718034040 tx=0x7fb7180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.811+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718032840 con 0x7fb7280c4f70 2024-03-23T15:52:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.812+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb718031f00 con 0x7fb7280c4f70 2024-03-23T15:52:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.812+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72815ba70 con 0x7fb7280c4f70 2024-03-23T15:52:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.812+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718032be0 con 0x7fb7280c4f70 2024-03-23T15:52:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.812+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb72814e620 con 0x7fb7280c4f70 2024-03-23T15:52:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.813+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb728075740 con 0x7fb7280c4f70 2024-03-23T15:52:02.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.832+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb718032d40 con 0x7fb7280c4f70 2024-03-23T15:52:02.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.833+0000 7fb70ffff640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb71406b860 0x7fb71406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:02.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.833+0000 7fb726ffd640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb71406b860 0x7fb71406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:02.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:02.833+0000 7fb726ffd640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb71406b860 0x7fb71406dd20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb728150520 tx=0x7fb71c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:03.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.082+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb71807a660 con 0x7fb7280c4f70 2024-03-23T15:52:03.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.082+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb718076900 con 0x7fb7280c4f70 2024-03-23T15:52:03.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.351+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7280c5da0 con 0x7fb7280c4f70 2024-03-23T15:52:03.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.365+0000 7fb70ffff640 1 -- 172.21.15.129:0/2872739411 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb718079fc0 con 0x7fb7280c4f70 2024-03-23T15:52:03.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:03.369 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:03.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb71406b860 msgr2=0x7fb71406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:03.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb71406b860 0x7fb71406dd20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb728150520 tx=0x7fb71c007d30 comp rx=0 tx=0).stop 2024-03-23T15:52:03.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 msgr2=0x7fb72815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:03.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 0x7fb72815bd00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb718034040 tx=0x7fb7180046c0 comp rx=0 tx=0).stop 2024-03-23T15:52:03.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 shutdown_connections 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb71406b860 0x7fb71406dd20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb72815ad10 0x7fb7281f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb728154270 0x7fb72814ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 --2- 172.21.15.129:0/2872739411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7280c4f70 0x7fb72815bd00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 >> 172.21.15.129:0/2872739411 conn(0x7fb72806d290 msgr2=0x7fb7280cbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 shutdown_connections 2024-03-23T15:52:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:03.368+0000 7fb72db3a640 1 -- 172.21.15.129:0/2872739411 wait complete. 2024-03-23T15:52:03.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:03.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:04.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:05.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.086+0000 7f2ab0d77640 1 Processor -- start 2024-03-23T15:52:05.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.086+0000 7f2ab0d77640 1 -- start start 2024-03-23T15:52:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.088+0000 7f2ab0d77640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2aac0ceb00 0x7f2aac15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.088+0000 7f2ab0d77640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2aac0cef40 0x7f2aac15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.088+0000 7f2ab0d77640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aac15b290 con 0x7f2aac0ccd60 2024-03-23T15:52:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.088+0000 7f2ab0d77640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f2aac0ce800 con 0x7f2aac0cef40 2024-03-23T15:52:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.088+0000 7f2ab0d77640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2aac154a70 con 0x7f2aac0ceb00 2024-03-23T15:52:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.089+0000 7f2aaad9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2aac0cef40 0x7f2aac15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.089+0000 7f2aa9d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2aac0ceb00 0x7f2aac15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.089+0000 7f2aaad9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2aac0cef40 0x7f2aac15b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42048/0 (socket says 172.21.15.129:42048) 2024-03-23T15:52:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.089+0000 7f2aaad9d640 1 -- 172.21.15.129:0/2814102893 learned_addr learned my addr 172.21.15.129:0/2814102893 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:05.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.092+0000 7f2aaad9d640 1 -- 172.21.15.129:0/2814102893 >> v2:172.21.15.157:3300/0 conn(0x7f2aac0ceb00 msgr2=0x7f2aac15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.092+0000 7f2aaad9d640 1 --2- 172.21.15.129:0/2814102893 >> v2:172.21.15.157:3300/0 conn(0x7f2aac0ceb00 0x7f2aac15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.092+0000 7f2aaad9d640 1 -- 172.21.15.129:0/2814102893 >> v1:172.21.15.157:6789/0 conn(0x7f2aac0ccd60 legacy=0x7f2aac06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.092+0000 7f2aaad9d640 1 -- 172.21.15.129:0/2814102893 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac0cd950 con 0x7f2aac0cef40 2024-03-23T15:52:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.094+0000 7f2aaad9d640 1 --2- 172.21.15.129:0/2814102893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac15b8f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2aac08d1a0 tx=0x7f2aa402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d153c182608e37 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:05.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.098+0000 7f2aa959a640 1 -- 172.21.15.129:0/2814102893 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa402cb20 con 0x7f2aac0cef40 2024-03-23T15:52:05.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.098+0000 7f2aa959a640 1 -- 172.21.15.129:0/2814102893 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa402cc80 con 0x7f2aac0cef40 2024-03-23T15:52:05.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.098+0000 7f2aa959a640 1 -- 172.21.15.129:0/2814102893 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40343f0 con 0x7f2aac0cef40 2024-03-23T15:52:05.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 -- 172.21.15.129:0/2814102893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 msgr2=0x7f2aac15b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:05.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/2814102893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac15b8f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2aac08d1a0 tx=0x7f2aa402bf80 comp rx=0 tx=0).stop 2024-03-23T15:52:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 -- 172.21.15.129:0/2814102893 shutdown_connections 2024-03-23T15:52:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/2814102893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac15b8f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/2814102893 >> v2:172.21.15.157:3300/0 conn(0x7f2aac0ceb00 0x7f2aac15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 -- 172.21.15.129:0/2814102893 >> 172.21.15.129:0/2814102893 conn(0x7f2aac0b1840 msgr2=0x7f2aac0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.099+0000 7f2ab0d77640 1 -- 172.21.15.129:0/2814102893 shutdown_connections 2024-03-23T15:52:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.100+0000 7f2ab0d77640 1 -- 172.21.15.129:0/2814102893 wait complete. 2024-03-23T15:52:05.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.101+0000 7f2ab0d77640 1 Processor -- start 2024-03-23T15:52:05.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.102+0000 7f2ab0d77640 1 -- start start 2024-03-23T15:52:05.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2ab0d77640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aac0ccd60 0x7f2aac1e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:05.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2ab0d77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2aac0ceb00 0x7f2aac1590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:05.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2ab0d77640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:05.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2ab0d77640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac1e5890 con 0x7f2aac0cef40 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2ab0d77640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac1e3bc0 con 0x7f2aac0ceb00 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2ab0d77640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2aac158c50 con 0x7f2aac0ccd60 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2aaad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2aa9d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2aac0ceb00 0x7f2aac1590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.103+0000 7f2aaa59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aac0ccd60 0x7f2aac1e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.104+0000 7f2aaad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac159620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42050/0 (socket says 172.21.15.129:42050) 2024-03-23T15:52:05.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.104+0000 7f2aaad9d640 1 -- 172.21.15.129:0/1286744057 learned_addr learned my addr 172.21.15.129:0/1286744057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:05.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.108+0000 7f2aaad9d640 1 -- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aac0ccd60 msgr2=0x7f2aac1e3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:05.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.108+0000 7f2aaad9d640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aac0ccd60 0x7f2aac1e3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.109+0000 7f2aaad9d640 1 -- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2aac0ceb00 msgr2=0x7f2aac1590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:05.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.109+0000 7f2aaad9d640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2aac0ceb00 0x7f2aac1590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.109+0000 7f2aaad9d640 1 -- 172.21.15.129:0/1286744057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa4009b30 con 0x7f2aac0cef40 2024-03-23T15:52:05.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.109+0000 7f2aa9d9b640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2aac0ceb00 0x7f2aac1590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:05.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.112+0000 7f2aaad9d640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac159620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2aa402c590 tx=0x7f2aa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.117+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4007e70 con 0x7f2aac0cef40 2024-03-23T15:52:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.117+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa4009510 con 0x7f2aac0cef40 2024-03-23T15:52:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.117+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa403e690 con 0x7f2aac0cef40 2024-03-23T15:52:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.117+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aac14bb80 con 0x7f2aac0cef40 2024-03-23T15:52:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.117+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aac15d320 con 0x7f2aac0cef40 2024-03-23T15:52:05.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.118+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aac04f530 con 0x7f2aac0cef40 2024-03-23T15:52:05.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.146+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2aa403e7f0 con 0x7f2aac0cef40 2024-03-23T15:52:05.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.147+0000 7f2a9affd640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2a8c06b860 0x7f2a8c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:05.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.147+0000 7f2aaa59c640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2a8c06b860 0x7f2a8c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:05.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.148+0000 7f2aaa59c640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2a8c06b860 0x7f2a8c06dd20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c006fd0 tx=0x7f2a9c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:05.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.153+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2aa407bc10 con 0x7f2aac0cef40 2024-03-23T15:52:05.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.400+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2aa4077eb0 con 0x7f2aac0cef40 2024-03-23T15:52:05.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.646+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2aac06db50 con 0x7f2aac0cef40 2024-03-23T15:52:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.660+0000 7f2a9affd640 1 -- 172.21.15.129:0/1286744057 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2aa407b570 con 0x7f2aac0cef40 2024-03-23T15:52:05.663 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:05.664 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2a8c06b860 msgr2=0x7f2a8c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2a8c06b860 0x7f2a8c06dd20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c006fd0 tx=0x7f2a9c009060 comp rx=0 tx=0).stop 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 msgr2=0x7f2aac159620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac159620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2aa402c590 tx=0x7f2aa4004a40 comp rx=0 tx=0).stop 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 shutdown_connections 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2aac0cef40 0x7f2aac159620 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2a8c06b860 0x7f2a8c06dd20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2aac0ceb00 0x7f2aac1590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 --2- 172.21.15.129:0/1286744057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aac0ccd60 0x7f2aac1e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 >> 172.21.15.129:0/1286744057 conn(0x7f2aac0b1840 msgr2=0x7f2aac1540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 shutdown_connections 2024-03-23T15:52:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:05.662+0000 7f2ab0d77640 1 -- 172.21.15.129:0/1286744057 wait complete. 2024-03-23T15:52:06.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:06.281 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:06.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:52:06.977 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:52:07.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:07.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.382+0000 7f8e4ccd4640 1 Processor -- start 2024-03-23T15:52:07.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.382+0000 7f8e4ccd4640 1 -- start start 2024-03-23T15:52:07.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.383+0000 7f8e4ccd4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8e480ccd60 0x7f8e4815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.383+0000 7f8e4ccd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480ceb00 0x7f8e4806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.383+0000 7f8e4ccd4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8e480cef40 0x7f8e4806c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.383+0000 7f8e4ccd4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8e4815b290 con 0x7f8e480ccd60 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.383+0000 7f8e4ccd4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8e480ce800 con 0x7f8e480cef40 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.383+0000 7f8e4ccd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8e48154a70 con 0x7f8e480ceb00 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.384+0000 7f8e4659c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8e480ccd60 0x7f8e4815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.384+0000 7f8e46d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8e480cef40 0x7f8e4806c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.384+0000 7f8e3ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480ceb00 0x7f8e4806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.384+0000 7f8e46d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8e480cef40 0x7f8e4806c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35048/0 (socket says 172.21.15.129:35048) 2024-03-23T15:52:07.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.384+0000 7f8e46d9d640 1 -- 172.21.15.129:0/989255818 learned_addr learned my addr 172.21.15.129:0/989255818 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:07.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.387+0000 7f8e46d9d640 1 -- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480ceb00 msgr2=0x7f8e4806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:07.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.387+0000 7f8e46d9d640 1 --2- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480ceb00 0x7f8e4806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.387+0000 7f8e46d9d640 1 -- 172.21.15.129:0/989255818 >> v2:172.21.15.129:3300/0 conn(0x7f8e480ccd60 msgr2=0x7f8e4815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:07.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.387+0000 7f8e46d9d640 1 --2- 172.21.15.129:0/989255818 >> v2:172.21.15.129:3300/0 conn(0x7f8e480ccd60 0x7f8e4815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.387+0000 7f8e46d9d640 1 -- 172.21.15.129:0/989255818 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e480cd950 con 0x7f8e480cef40 2024-03-23T15:52:07.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.387+0000 7f8e4659c640 1 --2- 172.21.15.129:0/989255818 >> v2:172.21.15.129:3300/0 conn(0x7f8e480ccd60 0x7f8e4815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:07.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.390+0000 7f8e46d9d640 1 --2- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480cef40 0x7f8e4806c7b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e4815e8e0 tx=0x7f8e4002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49de38a8913bec93 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:07.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.394+0000 7f8e45d9b640 1 -- 172.21.15.129:0/989255818 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4002cb00 con 0x7f8e480cef40 2024-03-23T15:52:07.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.394+0000 7f8e45d9b640 1 -- 172.21.15.129:0/989255818 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e4002cc60 con 0x7f8e480cef40 2024-03-23T15:52:07.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.394+0000 7f8e45d9b640 1 -- 172.21.15.129:0/989255818 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400343f0 con 0x7f8e480cef40 2024-03-23T15:52:07.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480cef40 msgr2=0x7f8e4806c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:07.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480cef40 0x7f8e4806c7b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e4815e8e0 tx=0x7f8e4002bf60 comp rx=0 tx=0).stop 2024-03-23T15:52:07.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/989255818 shutdown_connections 2024-03-23T15:52:07.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480cef40 0x7f8e4806c7b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/989255818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480ceb00 0x7f8e4806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/989255818 >> v2:172.21.15.129:3300/0 conn(0x7f8e480ccd60 0x7f8e4815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/989255818 >> 172.21.15.129:0/989255818 conn(0x7f8e480b1840 msgr2=0x7f8e480b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/989255818 shutdown_connections 2024-03-23T15:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.395+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/989255818 wait complete. 2024-03-23T15:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.396+0000 7f8e4ccd4640 1 Processor -- start 2024-03-23T15:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.396+0000 7f8e4ccd4640 1 -- start start 2024-03-23T15:52:07.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.397+0000 7f8e4ccd4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 0x7f8e481eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.397+0000 7f8e4ccd4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e480ceb00 0x7f8e481ef380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.397+0000 7f8e4ccd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480cef40 0x7f8e481f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.397+0000 7f8e4ccd4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e480b4670 con 0x7f8e480ceb00 2024-03-23T15:52:07.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.397+0000 7f8e4ccd4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8e480b4180 con 0x7f8e480ccd60 2024-03-23T15:52:07.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.397+0000 7f8e4ccd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8e481477f0 con 0x7f8e480cef40 2024-03-23T15:52:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.398+0000 7f8e3ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e480ceb00 0x7f8e481ef380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.398+0000 7f8e4659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 0x7f8e481eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.398+0000 7f8e46d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480cef40 0x7f8e481f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.398+0000 7f8e3ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e480ceb00 0x7f8e481ef380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42072/0 (socket says 172.21.15.129:42072) 2024-03-23T15:52:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.398+0000 7f8e4659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 0x7f8e481eee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35054/0 (socket says 172.21.15.129:35054) 2024-03-23T15:52:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.398+0000 7f8e3ffff640 1 -- 172.21.15.129:0/3397298757 learned_addr learned my addr 172.21.15.129:0/3397298757 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.401+0000 7f8e4659c640 1 -- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480cef40 msgr2=0x7f8e481f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.401+0000 7f8e4659c640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480cef40 0x7f8e481f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.401+0000 7f8e4659c640 1 -- 172.21.15.129:0/3397298757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e480ceb00 msgr2=0x7f8e481ef380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.401+0000 7f8e4659c640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e480ceb00 0x7f8e481ef380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.402+0000 7f8e4659c640 1 -- 172.21.15.129:0/3397298757 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e38009400 con 0x7f8e480ccd60 2024-03-23T15:52:07.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.404+0000 7f8e4659c640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 0x7f8e481eee40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800e8d0 tx=0x7f8e3800ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:07.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.408+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800c940 con 0x7f8e480ccd60 2024-03-23T15:52:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.408+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3800caa0 con 0x7f8e480ccd60 2024-03-23T15:52:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.408+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e40009b30 con 0x7f8e480ccd60 2024-03-23T15:52:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.408+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800ce50 con 0x7f8e480ccd60 2024-03-23T15:52:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.408+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e481495a0 con 0x7f8e480ccd60 2024-03-23T15:52:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.409+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4804f530 con 0x7f8e480ccd60 2024-03-23T15:52:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.428+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8e38013470 con 0x7f8e480ccd60 2024-03-23T15:52:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.429+0000 7f8e3effd640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8e2806b890 0x7f8e2806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.429+0000 7f8e3ffff640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8e2806b890 0x7f8e2806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.430+0000 7f8e3ffff640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8e2806b890 0x7f8e2806dd50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e480b47d0 tx=0x7f8e30009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.431+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8e38014e40 con 0x7f8e480ccd60 2024-03-23T15:52:07.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.690+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8e38055540 con 0x7f8e480ccd60 2024-03-23T15:52:07.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.910+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e4806db50 con 0x7f8e480ccd60 2024-03-23T15:52:07.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.924+0000 7f8e3effd640 1 -- 172.21.15.129:0/3397298757 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8e38021090 con 0x7f8e480ccd60 2024-03-23T15:52:07.928 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:07.928 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:07.928 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8e2806b890 msgr2=0x7f8e2806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8e2806b890 0x7f8e2806dd50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e480b47d0 tx=0x7f8e30009190 comp rx=0 tx=0).stop 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 msgr2=0x7f8e481eee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 0x7f8e481eee40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800e8d0 tx=0x7f8e3800ed90 comp rx=0 tx=0).stop 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 shutdown_connections 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8e480cef40 0x7f8e481f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e480ceb00 0x7f8e481ef380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8e2806b890 0x7f8e2806dd50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 --2- 172.21.15.129:0/3397298757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8e480ccd60 0x7f8e481eee40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:07.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 >> 172.21.15.129:0/3397298757 conn(0x7f8e480b1840 msgr2=0x7f8e480ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:07.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 shutdown_connections 2024-03-23T15:52:07.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:07.927+0000 7f8e4ccd4640 1 -- 172.21.15.129:0/3397298757 wait complete. 2024-03-23T15:52:08.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:08.546 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:09.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:09.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.645+0000 7fc750e6f640 1 Processor -- start 2024-03-23T15:52:09.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.646+0000 7fc750e6f640 1 -- start start 2024-03-23T15:52:09.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.646+0000 7fc750e6f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc74c15dba0 0x7fc74c06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:09.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.646+0000 7fc750e6f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc74c0cca30 0x7fc74c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.646+0000 7fc750e6f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c0c4c00 con 0x7fc74c15df80 2024-03-23T15:52:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.647+0000 7fc750e6f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc74c0ccea0 con 0x7fc74c0cca30 2024-03-23T15:52:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.647+0000 7fc750e6f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc74c06c240 con 0x7fc74c15dba0 2024-03-23T15:52:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.647+0000 7fc749d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc74c0cca30 0x7fc74c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.647+0000 7fc74a59c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc74c15dba0 0x7fc74c06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.648+0000 7fc749d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc74c0cca30 0x7fc74c06ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42082/0 (socket says 172.21.15.129:42082) 2024-03-23T15:52:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.648+0000 7fc749d9b640 1 -- 172.21.15.129:0/3962189688 learned_addr learned my addr 172.21.15.129:0/3962189688 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:09.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.651+0000 7fc749d9b640 1 -- 172.21.15.129:0/3962189688 >> v2:172.21.15.157:3300/0 conn(0x7fc74c15dba0 msgr2=0x7fc74c06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:09.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.651+0000 7fc749d9b640 1 --2- 172.21.15.129:0/3962189688 >> v2:172.21.15.157:3300/0 conn(0x7fc74c15dba0 0x7fc74c06cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:09.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.651+0000 7fc749d9b640 1 -- 172.21.15.129:0/3962189688 >> v1:172.21.15.157:6789/0 conn(0x7fc74c15df80 legacy=0x7fc74c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:09.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.651+0000 7fc749d9b640 1 -- 172.21.15.129:0/3962189688 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c0cbb20 con 0x7fc74c0cca30 2024-03-23T15:52:09.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.653+0000 7fc749d9b640 1 --2- 172.21.15.129:0/3962189688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c06ff30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc740009850 tx=0x7fc74002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9955763c0e374a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.658+0000 7fc74959a640 1 -- 172.21.15.129:0/3962189688 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740004230 con 0x7fc74c0cca30 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.658+0000 7fc74959a640 1 -- 172.21.15.129:0/3962189688 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc740004390 con 0x7fc74c0cca30 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.658+0000 7fc74959a640 1 -- 172.21.15.129:0/3962189688 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74002f830 con 0x7fc74c0cca30 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 -- 172.21.15.129:0/3962189688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 msgr2=0x7fc74c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 --2- 172.21.15.129:0/3962189688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c06ff30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc740009850 tx=0x7fc74002ba10 comp rx=0 tx=0).stop 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 -- 172.21.15.129:0/3962189688 shutdown_connections 2024-03-23T15:52:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 --2- 172.21.15.129:0/3962189688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c06ff30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 --2- 172.21.15.129:0/3962189688 >> v2:172.21.15.157:3300/0 conn(0x7fc74c15dba0 0x7fc74c06cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 -- 172.21.15.129:0/3962189688 >> 172.21.15.129:0/3962189688 conn(0x7fc74c071300 msgr2=0x7fc74c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 -- 172.21.15.129:0/3962189688 shutdown_connections 2024-03-23T15:52:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.659+0000 7fc750e6f640 1 -- 172.21.15.129:0/3962189688 wait complete. 2024-03-23T15:52:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.660+0000 7fc750e6f640 1 Processor -- start 2024-03-23T15:52:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.660+0000 7fc750e6f640 1 -- start start 2024-03-23T15:52:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.661+0000 7fc750e6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.661+0000 7fc750e6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc74c15dba0 0x7fc74c14f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.661+0000 7fc750e6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc74c15df80 0x7fc74c155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.661+0000 7fc750e6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c1f39e0 con 0x7fc74c0cca30 2024-03-23T15:52:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.661+0000 7fc750e6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c1f3f30 con 0x7fc74c15dba0 2024-03-23T15:52:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.661+0000 7fc750e6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc74c1f09e0 con 0x7fc74c15df80 2024-03-23T15:52:09.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.662+0000 7fc74a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:09.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.662+0000 7fc749d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc74c15dba0 0x7fc74c14f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:09.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.662+0000 7fc74ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc74c15df80 0x7fc74c155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:09.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.662+0000 7fc74a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c0cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42098/0 (socket says 172.21.15.129:42098) 2024-03-23T15:52:09.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.662+0000 7fc74a59c640 1 -- 172.21.15.129:0/1046725699 learned_addr learned my addr 172.21.15.129:0/1046725699 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:09.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.667+0000 7fc74a59c640 1 -- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc74c15df80 msgr2=0x7fc74c155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:09.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.667+0000 7fc74a59c640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc74c15df80 0x7fc74c155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:09.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.667+0000 7fc74a59c640 1 -- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc74c15dba0 msgr2=0x7fc74c14f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:09.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.667+0000 7fc74a59c640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc74c15dba0 0x7fc74c14f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:09.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.667+0000 7fc74a59c640 1 -- 172.21.15.129:0/1046725699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc740009460 con 0x7fc74c0cca30 2024-03-23T15:52:09.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.672+0000 7fc74a59c640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c0cd9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e9e0 tx=0x7fc73c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:09.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.676+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c009670 con 0x7fc74c0cca30 2024-03-23T15:52:09.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.676+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc73c00c9d0 con 0x7fc74c0cca30 2024-03-23T15:52:09.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.676+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c154e90 con 0x7fc74c0cca30 2024-03-23T15:52:09.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.676+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00cd80 con 0x7fc74c0cca30 2024-03-23T15:52:09.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.676+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74c1efc20 con 0x7fc74c0cca30 2024-03-23T15:52:09.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.677+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c0686f0 con 0x7fc74c0cca30 2024-03-23T15:52:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.711+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc73c013470 con 0x7fc74c0cca30 2024-03-23T15:52:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.712+0000 7fc73affd640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc72c06b810 0x7fc72c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.712+0000 7fc749d9b640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc72c06b810 0x7fc72c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.713+0000 7fc749d9b640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc72c06b810 0x7fc72c06dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc740035000 tx=0x7fc740008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:09.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.968+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc73c014e40 con 0x7fc74c0cca30 2024-03-23T15:52:09.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:09.968+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc73c055700 con 0x7fc74c0cca30 2024-03-23T15:52:10.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.187+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc74c075740 con 0x7fc74c0cca30 2024-03-23T15:52:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.201+0000 7fc73affd640 1 -- 172.21.15.129:0/1046725699 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc73c017090 con 0x7fc74c0cca30 2024-03-23T15:52:10.204 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:10.204 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:10.205 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc72c06b810 msgr2=0x7fc72c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc72c06b810 0x7fc72c06dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc740035000 tx=0x7fc740008030 comp rx=0 tx=0).stop 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 msgr2=0x7fc74c0cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c0cd9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e9e0 tx=0x7fc73c00eea0 comp rx=0 tx=0).stop 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 shutdown_connections 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc74c15df80 0x7fc74c155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc72c06b810 0x7fc72c06dcd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc74c15dba0 0x7fc74c14f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 --2- 172.21.15.129:0/1046725699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc74c0cca30 0x7fc74c0cd9c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 >> 172.21.15.129:0/1046725699 conn(0x7fc74c071300 msgr2=0x7fc74c1508e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 shutdown_connections 2024-03-23T15:52:10.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:10.203+0000 7fc750e6f640 1 -- 172.21.15.129:0/1046725699 wait complete. 2024-03-23T15:52:10.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:10.822 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:11.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.921+0000 7f3e28623640 1 Processor -- start 2024-03-23T15:52:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.921+0000 7f3e28623640 1 -- start start 2024-03-23T15:52:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.923+0000 7f3e28623640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3e200c38a0 0x7f3e2015b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.923+0000 7f3e28623640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e20158ce0 0x7f3e200c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.923+0000 7f3e28623640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e200ca020 0x7f3e200c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.923+0000 7f3e28623640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3e20158010 con 0x7f3e200ca020 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.923+0000 7f3e28623640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3e2015bb40 con 0x7f3e200c38a0 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.923+0000 7f3e28623640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e200c8420 con 0x7f3e20158ce0 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.924+0000 7f3e26bc0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e200ca020 0x7f3e200c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.924+0000 7f3e25bbe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e20158ce0 0x7f3e200c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:11.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.924+0000 7f3e26bc0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e200ca020 0x7f3e200c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40706/0 (socket says 172.21.15.129:40706) 2024-03-23T15:52:11.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.924+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1852893056 learned_addr learned my addr 172.21.15.129:0/1852893056 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.927+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1852893056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e20158ce0 msgr2=0x7f3e200c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.927+0000 7f3e26bc0640 1 --2- 172.21.15.129:0/1852893056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e20158ce0 0x7f3e200c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.927+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1852893056 >> v2:172.21.15.157:3300/0 conn(0x7f3e200c38a0 msgr2=0x7f3e2015b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:11.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.927+0000 7f3e26bc0640 1 --2- 172.21.15.129:0/1852893056 >> v2:172.21.15.157:3300/0 conn(0x7f3e200c38a0 0x7f3e2015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.927+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1852893056 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e200c7e60 con 0x7f3e200ca020 2024-03-23T15:52:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.930+0000 7f3e26bc0640 1 --2- 172.21.15.129:0/1852893056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200ca020 0x7f3e200c62e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e200c9ad0 tx=0x7f3e1802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc990640127242ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:11.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.934+0000 7f3e253bd640 1 -- 172.21.15.129:0/1852893056 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1802ca70 con 0x7f3e200ca020 2024-03-23T15:52:11.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.934+0000 7f3e253bd640 1 -- 172.21.15.129:0/1852893056 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e1802cbd0 con 0x7f3e200ca020 2024-03-23T15:52:11.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.934+0000 7f3e253bd640 1 -- 172.21.15.129:0/1852893056 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e180343f0 con 0x7f3e200ca020 2024-03-23T15:52:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 -- 172.21.15.129:0/1852893056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200ca020 msgr2=0x7f3e200c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 --2- 172.21.15.129:0/1852893056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200ca020 0x7f3e200c62e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e200c9ad0 tx=0x7f3e1802bed0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 -- 172.21.15.129:0/1852893056 shutdown_connections 2024-03-23T15:52:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 --2- 172.21.15.129:0/1852893056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200ca020 0x7f3e200c62e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 --2- 172.21.15.129:0/1852893056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e20158ce0 0x7f3e200c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 --2- 172.21.15.129:0/1852893056 >> v2:172.21.15.157:3300/0 conn(0x7f3e200c38a0 0x7f3e2015b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 -- 172.21.15.129:0/1852893056 >> 172.21.15.129:0/1852893056 conn(0x7f3e2006e230 msgr2=0x7f3e2006e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 -- 172.21.15.129:0/1852893056 shutdown_connections 2024-03-23T15:52:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.935+0000 7f3e28623640 1 -- 172.21.15.129:0/1852893056 wait complete. 2024-03-23T15:52:11.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.937+0000 7f3e28623640 1 Processor -- start 2024-03-23T15:52:11.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.937+0000 7f3e28623640 1 -- start start 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.938+0000 7f3e28623640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200c38a0 0x7f3e201f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.938+0000 7f3e28623640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e200ca020 0x7f3e2013a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.938+0000 7f3e28623640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e20158ce0 0x7f3e2013f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.938+0000 7f3e28623640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e201e0fa0 con 0x7f3e200c38a0 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.938+0000 7f3e28623640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3e201f21f0 con 0x7f3e20158ce0 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.938+0000 7f3e28623640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e2013e3c0 con 0x7f3e200ca020 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.939+0000 7f3e25bbe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e200ca020 0x7f3e2013a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.939+0000 7f3e26bc0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e20158ce0 0x7f3e2013f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:11.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.939+0000 7f3e25bbe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e200ca020 0x7f3e2013a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:33216/0 (socket says 172.21.15.129:33216) 2024-03-23T15:52:11.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.939+0000 7f3e25bbe640 1 -- 172.21.15.129:0/1656508424 learned_addr learned my addr 172.21.15.129:0/1656508424 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.944+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e200ca020 msgr2=0x7f3e2013a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.944+0000 7f3e26bc0640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e200ca020 0x7f3e2013a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.944+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1656508424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200c38a0 msgr2=0x7f3e201f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.944+0000 7f3e26bc0640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200c38a0 0x7f3e201f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.944+0000 7f3e26bc0640 1 -- 172.21.15.129:0/1656508424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e18009b30 con 0x7f3e20158ce0 2024-03-23T15:52:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.948+0000 7f3e26bc0640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e20158ce0 0x7f3e2013f5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e18032000 tx=0x7f3e18004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.952+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18009510 con 0x7f3e20158ce0 2024-03-23T15:52:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.952+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e1803e450 con 0x7f3e20158ce0 2024-03-23T15:52:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.952+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e20155700 con 0x7f3e20158ce0 2024-03-23T15:52:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.952+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1803e800 con 0x7f3e20158ce0 2024-03-23T15:52:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.952+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e2013e610 con 0x7f3e20158ce0 2024-03-23T15:52:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.953+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e20075740 con 0x7f3e20158ce0 2024-03-23T15:52:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.984+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e1803e960 con 0x7f3e20158ce0 2024-03-23T15:52:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.985+0000 7f3e16ffd640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0406b860 0x7f3e0406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.985+0000 7f3e263bf640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0406b860 0x7f3e0406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:11.985+0000 7f3e263bf640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0406b860 0x7f3e0406dd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009670 tx=0x7f3e1c006bf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:12.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.239+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3e1807c3a0 con 0x7f3e20158ce0 2024-03-23T15:52:12.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.263+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e18077f40 con 0x7f3e20158ce0 2024-03-23T15:52:12.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.489+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e20159a40 con 0x7f3e20158ce0 2024-03-23T15:52:12.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.511+0000 7f3e16ffd640 1 -- 172.21.15.129:0/1656508424 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3e1807b600 con 0x7f3e20158ce0 2024-03-23T15:52:12.515 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:12.515 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:12.515 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:12.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0406b860 msgr2=0x7f3e0406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0406b860 0x7f3e0406dd20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009670 tx=0x7f3e1c006bf0 comp rx=0 tx=0).stop 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e20158ce0 msgr2=0x7f3e2013f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e20158ce0 0x7f3e2013f5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e18032000 tx=0x7f3e18004a40 comp rx=0 tx=0).stop 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 shutdown_connections 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0406b860 0x7f3e0406dd20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e20158ce0 0x7f3e2013f5d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e200ca020 0x7f3e2013a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 --2- 172.21.15.129:0/1656508424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e200c38a0 0x7f3e201f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 >> 172.21.15.129:0/1656508424 conn(0x7f3e2006e230 msgr2=0x7f3e20072540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 shutdown_connections 2024-03-23T15:52:12.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:12.514+0000 7f3e28623640 1 -- 172.21.15.129:0/1656508424 wait complete. 2024-03-23T15:52:13.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:13.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:14.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:14.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.233+0000 7ff1ea32c640 1 Processor -- start 2024-03-23T15:52:14.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.233+0000 7ff1ea32c640 1 -- start start 2024-03-23T15:52:14.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1ea32c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e41544b0 0x7ff1e415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:14.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1ea32c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff1e40caa00 0x7ff1e406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1ea32c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e40cbcb0 con 0x7ff1e406c0b0 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1ea32c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff1e40c5be0 con 0x7ff1e40caa00 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1ea32c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e415b280 con 0x7ff1e41544b0 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1e88c9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff1e406c0b0 0x7ff1e40c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56560/0 (socket says 172.21.15.129:56560) 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1e88c9640 1 -- 172.21.15.129:0/627052319 learned_addr learned my addr 172.21.15.129:0/627052319 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1e3fff640 1 --2- 172.21.15.129:0/627052319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e41544b0 0x7ff1e415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.234+0000 7ff1e37fe640 1 --2- 172.21.15.129:0/627052319 >> v2:172.21.15.157:3300/0 conn(0x7ff1e40caa00 0x7ff1e406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:14.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.237+0000 7ff1e2ffd640 1 -- 172.21.15.129:0/627052319 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772797027 0 0) 0x7ff1e40cbcb0 con 0x7ff1e406c0b0 2024-03-23T15:52:14.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.237+0000 7ff1e2ffd640 1 -- 172.21.15.129:0/627052319 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1cc003520 con 0x7ff1e406c0b0 2024-03-23T15:52:14.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.237+0000 7ff1e3fff640 1 -- 172.21.15.129:0/627052319 >> v2:172.21.15.157:3300/0 conn(0x7ff1e40caa00 msgr2=0x7ff1e406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:14.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.237+0000 7ff1e3fff640 1 --2- 172.21.15.129:0/627052319 >> v2:172.21.15.157:3300/0 conn(0x7ff1e40caa00 0x7ff1e406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.237+0000 7ff1e3fff640 1 -- 172.21.15.129:0/627052319 >> v1:172.21.15.129:6789/0 conn(0x7ff1e406c0b0 legacy=0x7ff1e40c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:14.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.237+0000 7ff1e3fff640 1 -- 172.21.15.129:0/627052319 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e415a9e0 con 0x7ff1e41544b0 2024-03-23T15:52:14.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.240+0000 7ff1e3fff640 1 --2- 172.21.15.129:0/627052319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e41544b0 0x7ff1e415a2e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40097a0 tx=0x7ff1d402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=743d174c5c96a117 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:14.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.242+0000 7ff1e2ffd640 1 -- 172.21.15.129:0/627052319 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4030610 con 0x7ff1e41544b0 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1e2ffd640 1 -- 172.21.15.129:0/627052319 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d4030770 con 0x7ff1e41544b0 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1e2ffd640 1 -- 172.21.15.129:0/627052319 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4030ae0 con 0x7ff1e41544b0 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 -- 172.21.15.129:0/627052319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e41544b0 msgr2=0x7ff1e415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/627052319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e41544b0 0x7ff1e415a2e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40097a0 tx=0x7ff1d402b960 comp rx=0 tx=0).stop 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 -- 172.21.15.129:0/627052319 shutdown_connections 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/627052319 >> v2:172.21.15.157:3300/0 conn(0x7ff1e40caa00 0x7ff1e406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/627052319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e41544b0 0x7ff1e415a2e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 -- 172.21.15.129:0/627052319 >> 172.21.15.129:0/627052319 conn(0x7ff1e4061b50 msgr2=0x7ff1e4061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 -- 172.21.15.129:0/627052319 shutdown_connections 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.245+0000 7ff1ea32c640 1 -- 172.21.15.129:0/627052319 wait complete. 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.247+0000 7ff1ea32c640 1 Processor -- start 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.247+0000 7ff1ea32c640 1 -- start start 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1ea32c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e406c0b0 0x7ff1e40cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1ea32c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 0x7ff1e40c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1ea32c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1e41544b0 0x7ff1e4148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1ea32c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e40cfdc0 con 0x7ff1e40caa00 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1ea32c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41440e0 con 0x7ff1e41544b0 2024-03-23T15:52:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1ea32c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e40c8420 con 0x7ff1e406c0b0 2024-03-23T15:52:14.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1e37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 0x7ff1e40c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:14.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1e88c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1e41544b0 0x7ff1e4148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:14.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1e3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e406c0b0 0x7ff1e40cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:14.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1e37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 0x7ff1e40c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40730/0 (socket says 172.21.15.129:40730) 2024-03-23T15:52:14.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.248+0000 7ff1e37fe640 1 -- 172.21.15.129:0/1569144522 learned_addr learned my addr 172.21.15.129:0/1569144522 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:14.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.252+0000 7ff1e37fe640 1 -- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e406c0b0 msgr2=0x7ff1e40cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:14.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.252+0000 7ff1e37fe640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e406c0b0 0x7ff1e40cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.252+0000 7ff1e37fe640 1 -- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1e41544b0 msgr2=0x7ff1e4148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:14.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.252+0000 7ff1e37fe640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1e41544b0 0x7ff1e4148120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.252+0000 7ff1e37fe640 1 -- 172.21.15.129:0/1569144522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d4009400 con 0x7ff1e40caa00 2024-03-23T15:52:14.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.254+0000 7ff1e37fe640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 0x7ff1e40c96b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800e9b0 tx=0x7ff1d800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:14.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.258+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8009670 con 0x7ff1e40caa00 2024-03-23T15:52:14.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.258+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d800c9a0 con 0x7ff1e40caa00 2024-03-23T15:52:14.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.258+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e40c86a0 con 0x7ff1e40caa00 2024-03-23T15:52:14.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.258+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800cd50 con 0x7ff1e40caa00 2024-03-23T15:52:14.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.258+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e40d0500 con 0x7ff1e40caa00 2024-03-23T15:52:14.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.259+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e4075740 con 0x7ff1e40caa00 2024-03-23T15:52:14.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.278+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff1d8013470 con 0x7ff1e40caa00 2024-03-23T15:52:14.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.278+0000 7ff1e0ff9640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1d006b840 0x7ff1d006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:14.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.279+0000 7ff1e3fff640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1d006b840 0x7ff1d006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:14.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.279+0000 7ff1e3fff640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1d006b840 0x7ff1d006dd00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4036000 tx=0x7ff1d4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:14.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.282+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff1d8014e40 con 0x7ff1e40caa00 2024-03-23T15:52:14.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.528+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff1d8055640 con 0x7ff1e40caa00 2024-03-23T15:52:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.779+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1e406be00 con 0x7ff1e40caa00 2024-03-23T15:52:14.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.793+0000 7ff1e0ff9640 1 -- 172.21.15.129:0/1569144522 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff1d8021090 con 0x7ff1e40caa00 2024-03-23T15:52:14.796 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:14.797 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:14.797 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1d006b840 msgr2=0x7ff1d006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1d006b840 0x7ff1d006dd00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4036000 tx=0x7ff1d4008030 comp rx=0 tx=0).stop 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 msgr2=0x7ff1e40c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 0x7ff1e40c96b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800e9b0 tx=0x7ff1d800ee70 comp rx=0 tx=0).stop 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 shutdown_connections 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1d006b840 0x7ff1d006dd00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1e41544b0 0x7ff1e4148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e40caa00 0x7ff1e40c96b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.795+0000 7ff1ea32c640 1 --2- 172.21.15.129:0/1569144522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1e406c0b0 0x7ff1e40cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:14.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.796+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 >> 172.21.15.129:0/1569144522 conn(0x7ff1e4061b50 msgr2=0x7ff1e40b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:14.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.796+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 shutdown_connections 2024-03-23T15:52:14.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:14.796+0000 7ff1ea32c640 1 -- 172.21.15.129:0/1569144522 wait complete. 2024-03-23T15:52:15.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:15.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:16.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:16.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 Processor -- start 2024-03-23T15:52:16.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 -- start start 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ceb00 0x7f0ae415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ae40cef40 0x7f0ae415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae415b290 con 0x7f0ae40ccd60 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae40ce800 con 0x7f0ae40cef40 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.629+0000 7f0aeb234640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae4154a70 con 0x7f0ae40ceb00 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.630+0000 7f0ae97d1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ae40cef40 0x7f0ae415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.630+0000 7f0ae3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ceb00 0x7f0ae415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:16.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.630+0000 7f0ae97d1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ae40cef40 0x7f0ae415b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40734/0 (socket says 172.21.15.129:40734) 2024-03-23T15:52:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.630+0000 7f0ae97d1640 1 -- 172.21.15.129:0/4117713269 learned_addr learned my addr 172.21.15.129:0/4117713269 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.633+0000 7f0ae97d1640 1 -- 172.21.15.129:0/4117713269 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ceb00 msgr2=0x7f0ae415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.633+0000 7f0ae97d1640 1 --2- 172.21.15.129:0/4117713269 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ceb00 0x7f0ae415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.633+0000 7f0ae97d1640 1 -- 172.21.15.129:0/4117713269 >> v1:172.21.15.157:6789/0 conn(0x7f0ae40ccd60 legacy=0x7f0ae406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:16.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.633+0000 7f0ae97d1640 1 -- 172.21.15.129:0/4117713269 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae40cd950 con 0x7f0ae40cef40 2024-03-23T15:52:16.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.635+0000 7f0ae97d1640 1 --2- 172.21.15.129:0/4117713269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae415b8f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009e50 tx=0x7f0adc02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bdf244dbe4709f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:16.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.639+0000 7f0ae37fe640 1 -- 172.21.15.129:0/4117713269 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc02cb90 con 0x7f0ae40cef40 2024-03-23T15:52:16.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.639+0000 7f0ae37fe640 1 -- 172.21.15.129:0/4117713269 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc02ccf0 con 0x7f0ae40cef40 2024-03-23T15:52:16.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.639+0000 7f0ae37fe640 1 -- 172.21.15.129:0/4117713269 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc034610 con 0x7f0ae40cef40 2024-03-23T15:52:16.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 -- 172.21.15.129:0/4117713269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 msgr2=0x7f0ae415b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 --2- 172.21.15.129:0/4117713269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae415b8f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009e50 tx=0x7f0adc02bff0 comp rx=0 tx=0).stop 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 -- 172.21.15.129:0/4117713269 shutdown_connections 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 --2- 172.21.15.129:0/4117713269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae415b8f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 --2- 172.21.15.129:0/4117713269 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ceb00 0x7f0ae415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 -- 172.21.15.129:0/4117713269 >> 172.21.15.129:0/4117713269 conn(0x7f0ae40b1840 msgr2=0x7f0ae40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 -- 172.21.15.129:0/4117713269 shutdown_connections 2024-03-23T15:52:16.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.640+0000 7f0aeb234640 1 -- 172.21.15.129:0/4117713269 wait complete. 2024-03-23T15:52:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.642+0000 7f0aeb234640 1 Processor -- start 2024-03-23T15:52:16.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.642+0000 7f0aeb234640 1 -- start start 2024-03-23T15:52:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0aeb234640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ccd60 0x7f0ae41f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0aeb234640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ae40ceb00 0x7f0ae41f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0aeb234640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae4151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0aeb234640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40c4260 con 0x7f0ae40cef40 2024-03-23T15:52:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0aeb234640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae41f1a10 con 0x7f0ae40ceb00 2024-03-23T15:52:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0aeb234640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae41f1cb0 con 0x7f0ae40ccd60 2024-03-23T15:52:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0ae97d1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae4151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0ae8fd0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ccd60 0x7f0ae41f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0ae3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ae40ceb00 0x7f0ae41f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0ae97d1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae4151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40742/0 (socket says 172.21.15.129:40742) 2024-03-23T15:52:16.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.644+0000 7f0ae97d1640 1 -- 172.21.15.129:0/86481895 learned_addr learned my addr 172.21.15.129:0/86481895 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.648+0000 7f0ae97d1640 1 -- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ccd60 msgr2=0x7f0ae41f2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.648+0000 7f0ae97d1640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ccd60 0x7f0ae41f2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.648+0000 7f0ae97d1640 1 -- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ae40ceb00 msgr2=0x7f0ae41f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.648+0000 7f0ae97d1640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ae40ceb00 0x7f0ae41f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.648+0000 7f0ae97d1640 1 -- 172.21.15.129:0/86481895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0adc009b30 con 0x7f0ae40cef40 2024-03-23T15:52:16.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.650+0000 7f0ae97d1640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae4151c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009e50 tx=0x7f0adc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.653+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc009510 con 0x7f0ae40cef40 2024-03-23T15:52:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.653+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc03e450 con 0x7f0ae40cef40 2024-03-23T15:52:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.653+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae41f1f90 con 0x7f0ae40cef40 2024-03-23T15:52:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.653+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc03e800 con 0x7f0ae40cef40 2024-03-23T15:52:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.653+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41f2550 con 0x7f0ae40cef40 2024-03-23T15:52:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.655+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae404f530 con 0x7f0ae40cef40 2024-03-23T15:52:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.677+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0adc03e960 con 0x7f0ae40cef40 2024-03-23T15:52:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.677+0000 7f0ae17fa640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0ac806b840 0x7f0ac806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.677+0000 7f0ae8fd0640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0ac806b840 0x7f0ac806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.678+0000 7f0ae8fd0640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0ac806b840 0x7f0ac806dd00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80016e0 tx=0x7f0ad8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.681+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0adc07b840 con 0x7f0ae40cef40 2024-03-23T15:52:16.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:16.945+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0adc077ae0 con 0x7f0ae40cef40 2024-03-23T15:52:17.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.169+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ae406db50 con 0x7f0ae40cef40 2024-03-23T15:52:17.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.184+0000 7f0ae17fa640 1 -- 172.21.15.129:0/86481895 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0adc02ce60 con 0x7f0ae40cef40 2024-03-23T15:52:17.187 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:17.187 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:17.187 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:17.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0ac806b840 msgr2=0x7f0ac806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:17.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0ac806b840 0x7f0ac806dd00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80016e0 tx=0x7f0ad8009040 comp rx=0 tx=0).stop 2024-03-23T15:52:17.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 msgr2=0x7f0ae4151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae4151c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009e50 tx=0x7f0adc004a40 comp rx=0 tx=0).stop 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 shutdown_connections 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ae40cef40 0x7f0ae4151c90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0ac806b840 0x7f0ac806dd00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ae40ceb00 0x7f0ae41f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 --2- 172.21.15.129:0/86481895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ae40ccd60 0x7f0ae41f2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 >> 172.21.15.129:0/86481895 conn(0x7f0ae40b1840 msgr2=0x7f0ae4158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 shutdown_connections 2024-03-23T15:52:17.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:17.186+0000 7f0aeb234640 1 -- 172.21.15.129:0/86481895 wait complete. 2024-03-23T15:52:17.804 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:17.805 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:18.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.904+0000 7f18bf7f4640 1 Processor -- start 2024-03-23T15:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.904+0000 7f18bf7f4640 1 -- start start 2024-03-23T15:52:18.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.905+0000 7f18bf7f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815d9c0 0x7f18b806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:18.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.905+0000 7f18bf7f4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f18b815dda0 0x7f18b815e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:18.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.905+0000 7f18bf7f4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b80cb810 con 0x7f18b80ca4a0 2024-03-23T15:52:18.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.905+0000 7f18bf7f4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f18b815e770 con 0x7f18b815dda0 2024-03-23T15:52:18.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.905+0000 7f18bf7f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f18b80cc3b0 con 0x7f18b815d9c0 2024-03-23T15:52:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.906+0000 7f18bd590640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815d9c0 0x7f18b806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.906+0000 7f18bdd91640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f18b815dda0 0x7f18b815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.906+0000 7f18bcd8f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f18b80ca4a0 0x7f18b80cbca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56568/0 (socket says 172.21.15.129:56568) 2024-03-23T15:52:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.906+0000 7f18bcd8f640 1 -- 172.21.15.129:0/68660225 learned_addr learned my addr 172.21.15.129:0/68660225 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.909+0000 7f18affff640 1 -- 172.21.15.129:0/68660225 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3320274412 0 0) 0x7f18b80cb810 con 0x7f18b80ca4a0 2024-03-23T15:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.909+0000 7f18affff640 1 -- 172.21.15.129:0/68660225 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a4003520 con 0x7f18b80ca4a0 2024-03-23T15:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.909+0000 7f18bdd91640 1 -- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815d9c0 msgr2=0x7f18b806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.909+0000 7f18bdd91640 1 --2- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815d9c0 0x7f18b806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.909+0000 7f18bdd91640 1 -- 172.21.15.129:0/68660225 >> v1:172.21.15.129:6789/0 conn(0x7f18b80ca4a0 legacy=0x7f18b80cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.909+0000 7f18bdd91640 1 -- 172.21.15.129:0/68660225 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b80cac00 con 0x7f18b815dda0 2024-03-23T15:52:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.911+0000 7f18bdd91640 1 --2- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b815dda0 0x7f18b815e230 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009d90 tx=0x7f18b002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f644522425ec074 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.915+0000 7f18affff640 1 -- 172.21.15.129:0/68660225 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b002cc50 con 0x7f18b815dda0 2024-03-23T15:52:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.915+0000 7f18affff640 1 -- 172.21.15.129:0/68660225 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b002cdb0 con 0x7f18b815dda0 2024-03-23T15:52:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.915+0000 7f18affff640 1 -- 172.21.15.129:0/68660225 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0034660 con 0x7f18b815dda0 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 -- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b815dda0 msgr2=0x7f18b815e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b815dda0 0x7f18b815e230 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009d90 tx=0x7f18b002bea0 comp rx=0 tx=0).stop 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 -- 172.21.15.129:0/68660225 shutdown_connections 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b815dda0 0x7f18b815e230 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/68660225 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815d9c0 0x7f18b806c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 -- 172.21.15.129:0/68660225 >> 172.21.15.129:0/68660225 conn(0x7f18b80cfb20 msgr2=0x7f18b80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 -- 172.21.15.129:0/68660225 shutdown_connections 2024-03-23T15:52:18.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.916+0000 7f18bf7f4640 1 -- 172.21.15.129:0/68660225 wait complete. 2024-03-23T15:52:18.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.917+0000 7f18bf7f4640 1 Processor -- start 2024-03-23T15:52:18.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.917+0000 7f18bf7f4640 1 -- start start 2024-03-23T15:52:18.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bf7f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b80ca4a0 0x7f18b8156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:18.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bf7f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b815d9c0 0x7f18b8158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:18.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bf7f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 0x7f18b81f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:18.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bf7f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18b8159e50 con 0x7f18b815d9c0 2024-03-23T15:52:18.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bf7f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18b81f0660 con 0x7f18b80ca4a0 2024-03-23T15:52:18.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bf7f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f18b8151070 con 0x7f18b815dda0 2024-03-23T15:52:18.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bcd8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b815d9c0 0x7f18b8158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:18.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bdd91640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 0x7f18b81f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:18.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.918+0000 7f18bd590640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b80ca4a0 0x7f18b8156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:18.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.919+0000 7f18bdd91640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 0x7f18b81f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:33274/0 (socket says 172.21.15.129:33274) 2024-03-23T15:52:18.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.919+0000 7f18bdd91640 1 -- 172.21.15.129:0/1835312473 learned_addr learned my addr 172.21.15.129:0/1835312473 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:18.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.924+0000 7f18bdd91640 1 -- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b80ca4a0 msgr2=0x7f18b8156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:18.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.924+0000 7f18bdd91640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b80ca4a0 0x7f18b8156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:18.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.924+0000 7f18bdd91640 1 -- 172.21.15.129:0/1835312473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b815d9c0 msgr2=0x7f18b8158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:18.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.924+0000 7f18bdd91640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b815d9c0 0x7f18b8158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:18.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.924+0000 7f18bdd91640 1 -- 172.21.15.129:0/1835312473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b0009a70 con 0x7f18b815dda0 2024-03-23T15:52:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.927+0000 7f18bdd91640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 0x7f18b81f3190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18b0037040 tx=0x7f18b0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:18.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.935+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b003e610 con 0x7f18b815dda0 2024-03-23T15:52:18.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.935+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b003e770 con 0x7f18b815dda0 2024-03-23T15:52:18.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.935+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b003eb20 con 0x7f18b815dda0 2024-03-23T15:52:18.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.935+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b81f0af0 con 0x7f18b815dda0 2024-03-23T15:52:18.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.936+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b81efd30 con 0x7f18b815dda0 2024-03-23T15:52:18.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.937+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b8075740 con 0x7f18b815dda0 2024-03-23T15:52:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.969+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f18b003b050 con 0x7f18b815dda0 2024-03-23T15:52:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.969+0000 7f18adffb640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f189c06b890 0x7f189c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:18.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.969+0000 7f18bd590640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f189c06b890 0x7f189c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:18.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:18.970+0000 7f18bd590640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f189c06b890 0x7f189c06dd50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009b40 tx=0x7f18b4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.224+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f18b007b4f0 con 0x7f18b815dda0 2024-03-23T15:52:19.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.238+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f18b0077790 con 0x7f18b815dda0 2024-03-23T15:52:19.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.482+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18b81f07a0 con 0x7f18b815dda0 2024-03-23T15:52:19.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.497+0000 7f18adffb640 1 -- 172.21.15.129:0/1835312473 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f18b0039020 con 0x7f18b815dda0 2024-03-23T15:52:19.501 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:19.501 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:19.501 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f189c06b890 msgr2=0x7f189c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f189c06b890 0x7f189c06dd50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009b40 tx=0x7f18b4009190 comp rx=0 tx=0).stop 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 msgr2=0x7f18b81f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 0x7f18b81f3190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18b0037040 tx=0x7f18b0004a40 comp rx=0 tx=0).stop 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 shutdown_connections 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18b815dda0 0x7f18b81f3190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:19.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b815d9c0 0x7f18b8158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f189c06b890 0x7f189c06dd50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 --2- 172.21.15.129:0/1835312473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18b80ca4a0 0x7f18b8156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 >> 172.21.15.129:0/1835312473 conn(0x7f18b80cfb20 msgr2=0x7f18b80cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 shutdown_connections 2024-03-23T15:52:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:19.500+0000 7f18bf7f4640 1 -- 172.21.15.129:0/1835312473 wait complete. 2024-03-23T15:52:20.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:20.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:21.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:21.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.217+0000 7f903ee2a640 1 Processor -- start 2024-03-23T15:52:21.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.217+0000 7f903ee2a640 1 -- start start 2024-03-23T15:52:21.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.218+0000 7f903ee2a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f903806c030 0x7f903806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:21.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.218+0000 7f903ee2a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f90380cbc60 0x7f903815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:21.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.218+0000 7f903ee2a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f903806cae0 con 0x7f90380cc080 2024-03-23T15:52:21.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.218+0000 7f903ee2a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f90380cabc0 con 0x7f90380cbc60 2024-03-23T15:52:21.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.218+0000 7f903ee2a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90380caea0 con 0x7f903806c030 2024-03-23T15:52:21.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.219+0000 7f9037fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f90380cbc60 0x7f903815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:21.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.219+0000 7f903cbc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f903806c030 0x7f903806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:21.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.219+0000 7f9037fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f90380cbc60 0x7f903815d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54596/0 (socket says 172.21.15.129:54596) 2024-03-23T15:52:21.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.219+0000 7f9037fff640 1 -- 172.21.15.129:0/3678499986 learned_addr learned my addr 172.21.15.129:0/3678499986 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:21.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.221+0000 7f9037fff640 1 -- 172.21.15.129:0/3678499986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f903806c030 msgr2=0x7f903806c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:21.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.221+0000 7f9037fff640 1 --2- 172.21.15.129:0/3678499986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f903806c030 0x7f903806c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.221+0000 7f9037fff640 1 -- 172.21.15.129:0/3678499986 >> v1:172.21.15.129:6789/0 conn(0x7f90380cc080 legacy=0x7f903815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:21.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.221+0000 7f9037fff640 1 -- 172.21.15.129:0/3678499986 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90380cb180 con 0x7f90380cbc60 2024-03-23T15:52:21.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.225+0000 7f9037fff640 1 --2- 172.21.15.129:0/3678499986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90380cbc60 0x7f903815d900 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9024009870 tx=0x7f902402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15655f390f080913 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:21.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.229+0000 7f90377fe640 1 -- 172.21.15.129:0/3678499986 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240317c0 con 0x7f90380cbc60 2024-03-23T15:52:21.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.229+0000 7f90377fe640 1 -- 172.21.15.129:0/3678499986 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9024031920 con 0x7f90380cbc60 2024-03-23T15:52:21.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.229+0000 7f90377fe640 1 -- 172.21.15.129:0/3678499986 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024031cb0 con 0x7f90380cbc60 2024-03-23T15:52:21.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 -- 172.21.15.129:0/3678499986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90380cbc60 msgr2=0x7f903815d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:21.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3678499986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90380cbc60 0x7f903815d900 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9024009870 tx=0x7f902402b860 comp rx=0 tx=0).stop 2024-03-23T15:52:21.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 -- 172.21.15.129:0/3678499986 shutdown_connections 2024-03-23T15:52:21.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3678499986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90380cbc60 0x7f903815d900 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3678499986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f903806c030 0x7f903806c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 -- 172.21.15.129:0/3678499986 >> 172.21.15.129:0/3678499986 conn(0x7f90380cfb20 msgr2=0x7f90380cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:21.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 -- 172.21.15.129:0/3678499986 shutdown_connections 2024-03-23T15:52:21.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.230+0000 7f903ee2a640 1 -- 172.21.15.129:0/3678499986 wait complete. 2024-03-23T15:52:21.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.231+0000 7f903ee2a640 1 Processor -- start 2024-03-23T15:52:21.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.231+0000 7f903ee2a640 1 -- start start 2024-03-23T15:52:21.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.232+0000 7f903ee2a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 0x7f90380b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:21.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.232+0000 7f903ee2a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90380cbc60 0x7f903815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:21.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.232+0000 7f903ee2a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90380cc080 0x7f903815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:21.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.232+0000 7f903ee2a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f90381f4af0 con 0x7f903806c030 2024-03-23T15:52:21.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.232+0000 7f903ee2a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f903814b930 con 0x7f90380cbc60 2024-03-23T15:52:21.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.232+0000 7f903ee2a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90381f3f00 con 0x7f90380cc080 2024-03-23T15:52:21.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.233+0000 7f903cbc6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 0x7f90380b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.233+0000 7f903d3c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90380cc080 0x7f903815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.233+0000 7f9037fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90380cbc60 0x7f903815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.233+0000 7f903cbc6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 0x7f90380b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54604/0 (socket says 172.21.15.129:54604) 2024-03-23T15:52:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.233+0000 7f903cbc6640 1 -- 172.21.15.129:0/3946317649 learned_addr learned my addr 172.21.15.129:0/3946317649 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.236+0000 7f903cbc6640 1 -- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90380cc080 msgr2=0x7f903815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.236+0000 7f903cbc6640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90380cc080 0x7f903815cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.236+0000 7f903cbc6640 1 -- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90380cbc60 msgr2=0x7f903815c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.236+0000 7f903cbc6640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90380cbc60 0x7f903815c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.236+0000 7f903cbc6640 1 -- 172.21.15.129:0/3946317649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90240094d0 con 0x7f903806c030 2024-03-23T15:52:21.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.238+0000 7f903cbc6640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 0x7f90380b01c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f902c00e8a0 tx=0x7f902c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.243+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c0094f0 con 0x7f903806c030 2024-03-23T15:52:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.243+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f902c00c940 con 0x7f903806c030 2024-03-23T15:52:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.243+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c00ccf0 con 0x7f903806c030 2024-03-23T15:52:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.243+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9038157700 con 0x7f903806c030 2024-03-23T15:52:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.243+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90381f4430 con 0x7f903806c030 2024-03-23T15:52:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.244+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90380c8da0 con 0x7f903806c030 2024-03-23T15:52:21.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.267+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f902c013470 con 0x7f903806c030 2024-03-23T15:52:21.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.268+0000 7f90357fa640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f901c06b840 0x7f901c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:21.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.268+0000 7f9037fff640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f901c06b840 0x7f901c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:21.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.268+0000 7f9037fff640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f901c06b840 0x7f901c06dd00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9024036000 tx=0x7f902403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:21.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.271+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f902c014e40 con 0x7f903806c030 2024-03-23T15:52:21.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.542+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f902c055480 con 0x7f903806c030 2024-03-23T15:52:21.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.785+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9038075740 con 0x7f903806c030 2024-03-23T15:52:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.797+0000 7f90357fa640 1 -- 172.21.15.129:0/3946317649 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f902c021090 con 0x7f903806c030 2024-03-23T15:52:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:21.801 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.800+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f901c06b840 msgr2=0x7f901c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.800+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f901c06b840 0x7f901c06dd00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9024036000 tx=0x7f902403d040 comp rx=0 tx=0).stop 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.800+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 msgr2=0x7f90380b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.800+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 0x7f90380b01c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f902c00e8a0 tx=0x7f902c00ed60 comp rx=0 tx=0).stop 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.800+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 shutdown_connections 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90380cc080 0x7f903815cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f901c06b840 0x7f901c06dd00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90380cbc60 0x7f903815c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 --2- 172.21.15.129:0/3946317649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f903806c030 0x7f90380b01c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:21.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 >> 172.21.15.129:0/3946317649 conn(0x7f90380cfb20 msgr2=0x7f90380c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:21.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 shutdown_connections 2024-03-23T15:52:21.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:21.801+0000 7f903ee2a640 1 -- 172.21.15.129:0/3946317649 wait complete. 2024-03-23T15:52:22.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:22.421 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:23.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:23.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.520+0000 7f417435c640 1 Processor -- start 2024-03-23T15:52:23.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.520+0000 7f417435c640 1 -- start start 2024-03-23T15:52:23.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.522+0000 7f417435c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f416c15e010 0x7f416c165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:23.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.522+0000 7f417435c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f416c06c830 con 0x7f416c06c030 2024-03-23T15:52:23.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.522+0000 7f417435c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f416c0cc370 con 0x7f416c06c410 2024-03-23T15:52:23.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.522+0000 7f417435c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f416c06cae0 con 0x7f416c15e010 2024-03-23T15:52:23.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.522+0000 7f41728f9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f416c15e010 0x7f416c165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:23.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.523+0000 7f41718f7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f416c06c410 0x7f416c15d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:38134/0 (socket says 172.21.15.129:38134) 2024-03-23T15:52:23.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.523+0000 7f41718f7640 1 -- 172.21.15.129:0/3758916872 learned_addr learned my addr 172.21.15.129:0/3758916872 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:23.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.526+0000 7f41710f6640 1 -- 172.21.15.129:0/3758916872 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219533493 0 0) 0x7f416c0cc370 con 0x7f416c06c410 2024-03-23T15:52:23.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.526+0000 7f41710f6640 1 -- 172.21.15.129:0/3758916872 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4158003520 con 0x7f416c06c410 2024-03-23T15:52:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.528+0000 7f41728f9640 1 -- 172.21.15.129:0/3758916872 >> v1:172.21.15.157:6789/0 conn(0x7f416c06c410 legacy=0x7f416c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.528+0000 7f41728f9640 1 -- 172.21.15.129:0/3758916872 >> v1:172.21.15.129:6789/0 conn(0x7f416c06c030 legacy=0x7f416c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.528+0000 7f41728f9640 1 -- 172.21.15.129:0/3758916872 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416c0cabc0 con 0x7f416c15e010 2024-03-23T15:52:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.532+0000 7f41728f9640 1 --2- 172.21.15.129:0/3758916872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c15e010 0x7f416c165c60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4164009d90 tx=0x7f416402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e707cf01ca8c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:23.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.536+0000 7f41710f6640 1 -- 172.21.15.129:0/3758916872 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416402cd00 con 0x7f416c15e010 2024-03-23T15:52:23.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.536+0000 7f41710f6640 1 -- 172.21.15.129:0/3758916872 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416402ce60 con 0x7f416c15e010 2024-03-23T15:52:23.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.536+0000 7f41710f6640 1 -- 172.21.15.129:0/3758916872 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164034660 con 0x7f416c15e010 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 -- 172.21.15.129:0/3758916872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c15e010 msgr2=0x7f416c165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 --2- 172.21.15.129:0/3758916872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c15e010 0x7f416c165c60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4164009d90 tx=0x7f416402bf50 comp rx=0 tx=0).stop 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 -- 172.21.15.129:0/3758916872 shutdown_connections 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 --2- 172.21.15.129:0/3758916872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c15e010 0x7f416c165c60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 -- 172.21.15.129:0/3758916872 >> 172.21.15.129:0/3758916872 conn(0x7f416c0cfb20 msgr2=0x7f416c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 -- 172.21.15.129:0/3758916872 shutdown_connections 2024-03-23T15:52:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.537+0000 7f417435c640 1 -- 172.21.15.129:0/3758916872 wait complete. 2024-03-23T15:52:23.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.538+0000 7f417435c640 1 Processor -- start 2024-03-23T15:52:23.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.538+0000 7f417435c640 1 -- start start 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f417435c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 0x7f416c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f417435c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f416c06c410 0x7f416c1f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f417435c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f416c15e010 0x7f416c1f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f417435c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f416c15ade0 con 0x7f416c06c030 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f417435c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f416c158310 con 0x7f416c15e010 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f417435c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f416c1585f0 con 0x7f416c06c410 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.539+0000 7f41720f8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 0x7f416c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.540+0000 7f41728f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f416c15e010 0x7f416c1f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.540+0000 7f41718f7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f416c06c410 0x7f416c1f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.540+0000 7f41720f8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 0x7f416c157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54614/0 (socket says 172.21.15.129:54614) 2024-03-23T15:52:23.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.540+0000 7f41720f8640 1 -- 172.21.15.129:0/3470018443 learned_addr learned my addr 172.21.15.129:0/3470018443 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.543+0000 7f41720f8640 1 -- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f416c06c410 msgr2=0x7f416c1f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.543+0000 7f41720f8640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f416c06c410 0x7f416c1f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.543+0000 7f41720f8640 1 -- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f416c15e010 msgr2=0x7f416c1f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.543+0000 7f41720f8640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f416c15e010 0x7f416c1f06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.543+0000 7f41720f8640 1 -- 172.21.15.129:0/3470018443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4164009a70 con 0x7f416c06c030 2024-03-23T15:52:23.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.545+0000 7f41720f8640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 0x7f416c157140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f416800c3c0 tx=0x7f416800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:23.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.549+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41680052d0 con 0x7f416c06c030 2024-03-23T15:52:23.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.549+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4168005450 con 0x7f416c06c030 2024-03-23T15:52:23.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.549+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168007260 con 0x7f416c06c030 2024-03-23T15:52:23.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.549+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f416c1588d0 con 0x7f416c06c030 2024-03-23T15:52:23.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.549+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f416c157680 con 0x7f416c06c030 2024-03-23T15:52:23.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.550+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f416c0686f0 con 0x7f416c06c030 2024-03-23T15:52:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.579+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f41680073c0 con 0x7f416c06c030 2024-03-23T15:52:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.580+0000 7f4162ffd640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f415006b890 0x7f415006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.580+0000 7f41718f7640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f415006b890 0x7f415006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.580+0000 7f41718f7640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f415006b890 0x7f415006dd50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f416c068470 tx=0x7f415c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:23.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.585+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f41680584c0 con 0x7f416c06c030 2024-03-23T15:52:23.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:23.833+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4168054760 con 0x7f416c06c030 2024-03-23T15:52:24.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.057+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f416c075740 con 0x7f416c06c030 2024-03-23T15:52:24.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.072+0000 7f4162ffd640 1 -- 172.21.15.129:0/3470018443 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4168020020 con 0x7f416c06c030 2024-03-23T15:52:24.076 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:24.076 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f415006b890 msgr2=0x7f415006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f415006b890 0x7f415006dd50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f416c068470 tx=0x7f415c007d30 comp rx=0 tx=0).stop 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 msgr2=0x7f416c157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 0x7f416c157140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f416800c3c0 tx=0x7f416800c880 comp rx=0 tx=0).stop 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 shutdown_connections 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f415006b890 0x7f415006dd50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.074+0000 7f417435c640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f416c15e010 0x7f416c1f06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:24.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.075+0000 7f417435c640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f416c06c410 0x7f416c1f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.075+0000 7f417435c640 1 --2- 172.21.15.129:0/3470018443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f416c06c030 0x7f416c157140 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.075+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 >> 172.21.15.129:0/3470018443 conn(0x7f416c0cfb20 msgr2=0x7f416c0cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.075+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 shutdown_connections 2024-03-23T15:52:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:24.075+0000 7f417435c640 1 -- 172.21.15.129:0/3470018443 wait complete. 2024-03-23T15:52:24.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:24.693 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:25.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.794+0000 7f87c23a2640 1 Processor -- start 2024-03-23T15:52:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.794+0000 7f87c23a2640 1 -- start start 2024-03-23T15:52:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.795+0000 7f87c23a2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f87bc154430 0x7f87bc0c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.795+0000 7f87c23a2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f87bc15abc0 0x7f87bc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.795+0000 7f87c23a2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc15e6c0 con 0x7f87bc0ca9a0 2024-03-23T15:52:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.795+0000 7f87c23a2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc15db90 con 0x7f87bc154430 2024-03-23T15:52:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.795+0000 7f87c23a2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc0c4a60 con 0x7f87bc15abc0 2024-03-23T15:52:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.796+0000 7f87c13a0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f87bc154430 0x7f87bc0c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.796+0000 7f87c1ba1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f87bc15abc0 0x7f87bc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:25.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.796+0000 7f87c0b9f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f87bc0ca9a0 0x7f87bc15dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:41150/0 (socket says 172.21.15.129:41150) 2024-03-23T15:52:25.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.796+0000 7f87c0b9f640 1 -- 172.21.15.129:0/2653814367 learned_addr learned my addr 172.21.15.129:0/2653814367 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.799+0000 7f87c1ba1640 1 -- 172.21.15.129:0/2653814367 >> v2:172.21.15.129:3300/0 conn(0x7f87bc154430 msgr2=0x7f87bc0c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.799+0000 7f87c1ba1640 1 --2- 172.21.15.129:0/2653814367 >> v2:172.21.15.129:3300/0 conn(0x7f87bc154430 0x7f87bc0c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.799+0000 7f87c1ba1640 1 -- 172.21.15.129:0/2653814367 >> v1:172.21.15.129:6789/0 conn(0x7f87bc0ca9a0 legacy=0x7f87bc15dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.799+0000 7f87c1ba1640 1 -- 172.21.15.129:0/2653814367 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc0c4d00 con 0x7f87bc15abc0 2024-03-23T15:52:25.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.801+0000 7f87c1ba1640 1 --2- 172.21.15.129:0/2653814367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc0cbe80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0ccf00 tx=0x7f87b402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d28e03cb8922353 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:25.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.806+0000 7f87b3fff640 1 -- 172.21.15.129:0/2653814367 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b402cbf0 con 0x7f87bc15abc0 2024-03-23T15:52:25.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.806+0000 7f87b3fff640 1 -- 172.21.15.129:0/2653814367 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b402cd50 con 0x7f87bc15abc0 2024-03-23T15:52:25.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.806+0000 7f87b3fff640 1 -- 172.21.15.129:0/2653814367 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4034660 con 0x7f87bc15abc0 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 -- 172.21.15.129:0/2653814367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 msgr2=0x7f87bc0cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 --2- 172.21.15.129:0/2653814367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc0cbe80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0ccf00 tx=0x7f87b402be40 comp rx=0 tx=0).stop 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 -- 172.21.15.129:0/2653814367 shutdown_connections 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 --2- 172.21.15.129:0/2653814367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc0cbe80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 --2- 172.21.15.129:0/2653814367 >> v2:172.21.15.129:3300/0 conn(0x7f87bc154430 0x7f87bc0c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 -- 172.21.15.129:0/2653814367 >> 172.21.15.129:0/2653814367 conn(0x7f87bc06c370 msgr2=0x7f87bc06c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 -- 172.21.15.129:0/2653814367 shutdown_connections 2024-03-23T15:52:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.807+0000 7f87c23a2640 1 -- 172.21.15.129:0/2653814367 wait complete. 2024-03-23T15:52:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.808+0000 7f87c23a2640 1 Processor -- start 2024-03-23T15:52:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.808+0000 7f87c23a2640 1 -- start start 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c23a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc0ca9a0 0x7f87bc1f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c23a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 0x7f87bc155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c23a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc14a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c23a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc14ac90 con 0x7f87bc154430 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c23a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc0c8f40 con 0x7f87bc15abc0 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c23a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc0c9220 con 0x7f87bc0ca9a0 2024-03-23T15:52:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.809+0000 7f87c0b9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 0x7f87bc155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:25.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.810+0000 7f87c1ba1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc14a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:25.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.810+0000 7f87c13a0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc0ca9a0 0x7f87bc1f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:25.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.810+0000 7f87c0b9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 0x7f87bc155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54642/0 (socket says 172.21.15.129:54642) 2024-03-23T15:52:25.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.810+0000 7f87c0b9f640 1 -- 172.21.15.129:0/1394096308 learned_addr learned my addr 172.21.15.129:0/1394096308 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:25.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.813+0000 7f87c0b9f640 1 -- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc0ca9a0 msgr2=0x7f87bc1f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:25.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.813+0000 7f87c0b9f640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc0ca9a0 0x7f87bc1f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:25.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.813+0000 7f87c0b9f640 1 -- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 msgr2=0x7f87bc14a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:25.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.813+0000 7f87c0b9f640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc14a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:25.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.813+0000 7f87c0b9f640 1 -- 172.21.15.129:0/1394096308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b4009a70 con 0x7f87bc154430 2024-03-23T15:52:25.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.816+0000 7f87c0b9f640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 0x7f87bc155020 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00c3c0 tx=0x7f87ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:25.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.819+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac005ce0 con 0x7f87bc154430 2024-03-23T15:52:25.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.819+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ac005e40 con 0x7f87bc154430 2024-03-23T15:52:25.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.819+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc0c9500 con 0x7f87bc154430 2024-03-23T15:52:25.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.820+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac0072d0 con 0x7f87bc154430 2024-03-23T15:52:25.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.820+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc1f04b0 con 0x7f87bc154430 2024-03-23T15:52:25.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:25.821+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc157a20 con 0x7f87bc154430 2024-03-23T15:52:26.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.074+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f87ac0074f0 con 0x7f87bc154430 2024-03-23T15:52:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.074+0000 7f87b1ffb640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f87a006b860 0x7f87a006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.074+0000 7f87c13a0640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f87a006b860 0x7f87a006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.075+0000 7f87c13a0640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f87a006b860 0x7f87a006dd20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f87b8009aa0 tx=0x7f87b8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:26.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.087+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f87ac0581b0 con 0x7f87bc154430 2024-03-23T15:52:26.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.087+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f87ac054450 con 0x7f87bc154430 2024-03-23T15:52:26.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.307+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87bc15e6c0 con 0x7f87bc154430 2024-03-23T15:52:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.322+0000 7f87b1ffb640 1 -- 172.21.15.129:0/1394096308 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f87bc15e6c0 con 0x7f87bc154430 2024-03-23T15:52:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:26.325 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f87a006b860 msgr2=0x7f87a006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f87a006b860 0x7f87a006dd20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f87b8009aa0 tx=0x7f87b8009190 comp rx=0 tx=0).stop 2024-03-23T15:52:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 msgr2=0x7f87bc155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 0x7f87bc155020 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00c3c0 tx=0x7f87ac00c880 comp rx=0 tx=0).stop 2024-03-23T15:52:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 shutdown_connections 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f87a006b860 0x7f87a006dd20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc15abc0 0x7f87bc14a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87bc154430 0x7f87bc155020 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 --2- 172.21.15.129:0/1394096308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc0ca9a0 0x7f87bc1f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 >> 172.21.15.129:0/1394096308 conn(0x7f87bc06c370 msgr2=0x7f87bc1526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 shutdown_connections 2024-03-23T15:52:26.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:26.324+0000 7f87c23a2640 1 -- 172.21.15.129:0/1394096308 wait complete. 2024-03-23T15:52:26.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:26.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:27.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:28.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.044+0000 7fd00bdb5640 1 Processor -- start 2024-03-23T15:52:28.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.045+0000 7fd00bdb5640 1 -- start start 2024-03-23T15:52:28.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.045+0000 7fd00bdb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0040cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:28.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.045+0000 7fd00bdb5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd00415df80 0x7fd0041540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.045+0000 7fd00bdb5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd004070640 con 0x7fd00415dba0 2024-03-23T15:52:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.045+0000 7fd00bdb5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd0040c4c00 con 0x7fd00415df80 2024-03-23T15:52:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.045+0000 7fd00bdb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd00406c240 con 0x7fd00406cf70 2024-03-23T15:52:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.046+0000 7fd009350640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0040cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.047+0000 7fd009350640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0040cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44236/0 (socket says 172.21.15.129:44236) 2024-03-23T15:52:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.047+0000 7fd009350640 1 -- 172.21.15.129:0/2211388364 learned_addr learned my addr 172.21.15.129:0/2211388364 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.048+0000 7fd00a352640 1 --2- 172.21.15.129:0/2211388364 >> v2:172.21.15.129:3300/0 conn(0x7fd00415df80 0x7fd0041540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.049+0000 7fd009350640 1 -- 172.21.15.129:0/2211388364 >> v2:172.21.15.129:3300/0 conn(0x7fd00415df80 msgr2=0x7fd0041540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.049+0000 7fd009350640 1 --2- 172.21.15.129:0/2211388364 >> v2:172.21.15.129:3300/0 conn(0x7fd00415df80 0x7fd0041540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.049+0000 7fd009350640 1 -- 172.21.15.129:0/2211388364 >> v1:172.21.15.129:6789/0 conn(0x7fd00415dba0 legacy=0x7fd00406ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.049+0000 7fd009350640 1 -- 172.21.15.129:0/2211388364 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0040cbb20 con 0x7fd00406cf70 2024-03-23T15:52:28.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.051+0000 7fd009350640 1 --2- 172.21.15.129:0/2211388364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0040cca30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009850 tx=0x7fcff402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ead3b62e37f9ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd008b4f640 1 -- 172.21.15.129:0/2211388364 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4004230 con 0x7fd00406cf70 2024-03-23T15:52:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd008b4f640 1 -- 172.21.15.129:0/2211388364 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff4004390 con 0x7fd00406cf70 2024-03-23T15:52:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd008b4f640 1 -- 172.21.15.129:0/2211388364 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4003810 con 0x7fd00406cf70 2024-03-23T15:52:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd00bdb5640 1 -- 172.21.15.129:0/2211388364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 msgr2=0x7fd0040cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/2211388364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0040cca30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009850 tx=0x7fcff402b960 comp rx=0 tx=0).stop 2024-03-23T15:52:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd00bdb5640 1 -- 172.21.15.129:0/2211388364 shutdown_connections 2024-03-23T15:52:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/2211388364 >> v2:172.21.15.129:3300/0 conn(0x7fd00415df80 0x7fd0041540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/2211388364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0040cca30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.056+0000 7fd00bdb5640 1 -- 172.21.15.129:0/2211388364 >> 172.21.15.129:0/2211388364 conn(0x7fd004071300 msgr2=0x7fd0040716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.057+0000 7fd00bdb5640 1 -- 172.21.15.129:0/2211388364 shutdown_connections 2024-03-23T15:52:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.057+0000 7fd00bdb5640 1 -- 172.21.15.129:0/2211388364 wait complete. 2024-03-23T15:52:28.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 Processor -- start 2024-03-23T15:52:28.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 -- start start 2024-03-23T15:52:28.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0041523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:28.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 0x7fd00414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:28.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd00415df80 0x7fd0041e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:28.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd00414ae60 con 0x7fd00415dba0 2024-03-23T15:52:28.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd0041f05e0 con 0x7fd00415df80 2024-03-23T15:52:28.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.058+0000 7fd00bdb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd0041f08a0 con 0x7fd00406cf70 2024-03-23T15:52:28.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.059+0000 7fd009350640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 0x7fd00414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.059+0000 7fd009b51640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0041523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.059+0000 7fd00a352640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd00415df80 0x7fd0041e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.059+0000 7fd009350640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 0x7fd00414b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54650/0 (socket says 172.21.15.129:54650) 2024-03-23T15:52:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.059+0000 7fd009350640 1 -- 172.21.15.129:0/1194533566 learned_addr learned my addr 172.21.15.129:0/1194533566 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.063+0000 7fd009350640 1 -- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 msgr2=0x7fd0041523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.063+0000 7fd009350640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0041523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.063+0000 7fd009350640 1 -- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd00415df80 msgr2=0x7fd0041e59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.063+0000 7fd009350640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd00415df80 0x7fd0041e59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.063+0000 7fd009350640 1 -- 172.21.15.129:0/1194533566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff4009460 con 0x7fd00415dba0 2024-03-23T15:52:28.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.065+0000 7fd009350640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 0x7fd00414b2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcff4000c00 tx=0x7fcff40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.069+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff403e070 con 0x7fd00415dba0 2024-03-23T15:52:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.069+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff4004500 con 0x7fd00415dba0 2024-03-23T15:52:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.069+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0041f0b80 con 0x7fd00415dba0 2024-03-23T15:52:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.069+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4003e10 con 0x7fd00415dba0 2024-03-23T15:52:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.069+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00414aaf0 con 0x7fd00415dba0 2024-03-23T15:52:28.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.070+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0040686f0 con 0x7fd00415dba0 2024-03-23T15:52:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.347+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcff4006040 con 0x7fd00415dba0 2024-03-23T15:52:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.348+0000 7fcffa7fc640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcfe806b840 0x7fcfe806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.348+0000 7fd009b51640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcfe806b840 0x7fcfe806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.349+0000 7fd009b51640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcfe806b840 0x7fcfe806dd00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd0000017f0 tx=0x7fd000009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:28.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.361+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcff407ab60 con 0x7fd00415dba0 2024-03-23T15:52:28.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.361+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcff4076e00 con 0x7fd00415dba0 2024-03-23T15:52:28.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.584+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd00414ac80 con 0x7fd00415dba0 2024-03-23T15:52:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.616+0000 7fcffa7fc640 1 -- 172.21.15.129:0/1194533566 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcff407a4c0 con 0x7fd00415dba0 2024-03-23T15:52:28.620 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:28.620 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcfe806b840 msgr2=0x7fcfe806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcfe806b840 0x7fcfe806dd00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd0000017f0 tx=0x7fd000009040 comp rx=0 tx=0).stop 2024-03-23T15:52:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 msgr2=0x7fd00414b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 0x7fd00414b2f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcff4000c00 tx=0x7fcff40046c0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 shutdown_connections 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcfe806b840 0x7fcfe806dd00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd00415df80 0x7fd0041e59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd00415dba0 0x7fd00414b2f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 --2- 172.21.15.129:0/1194533566 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd00406cf70 0x7fd0041523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 >> 172.21.15.129:0/1194533566 conn(0x7fd004071300 msgr2=0x7fd004153750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 shutdown_connections 2024-03-23T15:52:28.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:28.619+0000 7fd00bdb5640 1 -- 172.21.15.129:0/1194533566 wait complete. 2024-03-23T15:52:29.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:29.238 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:30.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:30.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.338+0000 7fedbc702640 1 Processor -- start 2024-03-23T15:52:30.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.338+0000 7fedbc702640 1 -- start start 2024-03-23T15:52:30.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.339+0000 7fedbc702640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:30.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.339+0000 7fedbc702640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fedb40cc080 0x7fedb415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:30.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.339+0000 7fedbc702640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb406cae0 con 0x7fedb40cbc60 2024-03-23T15:52:30.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.339+0000 7fedbc702640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fedb40cabc0 con 0x7fedb40cc080 2024-03-23T15:52:30.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.339+0000 7fedbc702640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fedb40caea0 con 0x7fedb406c030 2024-03-23T15:52:30.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.340+0000 7fedbac9f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fedb40cc080 0x7fedb415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:30.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.340+0000 7fedba49e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:30.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.340+0000 7fedb9c9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fedb40cbc60 0x7fedb415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:38142/0 (socket says 172.21.15.129:38142) 2024-03-23T15:52:30.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.340+0000 7fedb9c9d640 1 -- 172.21.15.129:0/774146941 learned_addr learned my addr 172.21.15.129:0/774146941 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:30.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.343+0000 7fedbac9f640 1 -- 172.21.15.129:0/774146941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 msgr2=0x7fedb406c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:30.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.343+0000 7fedbac9f640 1 --2- 172.21.15.129:0/774146941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb406c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.343+0000 7fedbac9f640 1 -- 172.21.15.129:0/774146941 >> v1:172.21.15.157:6789/0 conn(0x7fedb40cbc60 legacy=0x7fedb415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:30.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.343+0000 7fedbac9f640 1 -- 172.21.15.129:0/774146941 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb40cb180 con 0x7fedb40cc080 2024-03-23T15:52:30.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.345+0000 7fedbac9f640 1 --2- 172.21.15.129:0/774146941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cc080 0x7fedb415e010 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fedac009d90 tx=0x7fedac02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63bd5d8cbbdaa70 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:30.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.350+0000 7fedb949c640 1 -- 172.21.15.129:0/774146941 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac02cbc0 con 0x7fedb40cc080 2024-03-23T15:52:30.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.350+0000 7fedb949c640 1 -- 172.21.15.129:0/774146941 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedac02cd20 con 0x7fedb40cc080 2024-03-23T15:52:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.350+0000 7fedb949c640 1 -- 172.21.15.129:0/774146941 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac034660 con 0x7fedb40cc080 2024-03-23T15:52:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 -- 172.21.15.129:0/774146941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cc080 msgr2=0x7fedb415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 --2- 172.21.15.129:0/774146941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cc080 0x7fedb415e010 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fedac009d90 tx=0x7fedac02be10 comp rx=0 tx=0).stop 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 -- 172.21.15.129:0/774146941 shutdown_connections 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 --2- 172.21.15.129:0/774146941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cc080 0x7fedb415e010 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 --2- 172.21.15.129:0/774146941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb406c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 -- 172.21.15.129:0/774146941 >> 172.21.15.129:0/774146941 conn(0x7fedb40cfb20 msgr2=0x7fedb40cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 -- 172.21.15.129:0/774146941 shutdown_connections 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.351+0000 7fedbc702640 1 -- 172.21.15.129:0/774146941 wait complete. 2024-03-23T15:52:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 Processor -- start 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 -- start start 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb41560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cbc60 0x7fedb40b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fedb40cc080 0x7fedb40b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41f4310 con 0x7fedb40cbc60 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41efe50 con 0x7fedb40cc080 2024-03-23T15:52:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.353+0000 7fedbc702640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fedb41f0060 con 0x7fedb406c030 2024-03-23T15:52:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.354+0000 7fedb9c9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cbc60 0x7fedb40b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.354+0000 7fedba49e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb41560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.355+0000 7fedba49e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb41560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44262/0 (socket says 172.21.15.129:44262) 2024-03-23T15:52:30.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.355+0000 7fedba49e640 1 -- 172.21.15.129:0/578518652 learned_addr learned my addr 172.21.15.129:0/578518652 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:30.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.355+0000 7fedbac9f640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fedb40cc080 0x7fedb40b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.359+0000 7fedb9c9d640 1 -- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 msgr2=0x7fedb41560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.359+0000 7fedb9c9d640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb41560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.359+0000 7fedb9c9d640 1 -- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fedb40cc080 msgr2=0x7fedb40b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.359+0000 7fedb9c9d640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fedb40cc080 0x7fedb40b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.359+0000 7fedb9c9d640 1 -- 172.21.15.129:0/578518652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedac009a70 con 0x7fedb40cbc60 2024-03-23T15:52:30.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.362+0000 7fedb9c9d640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cbc60 0x7fedb40b28f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feda4000f60 tx=0x7feda400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:30.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.365+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4005ce0 con 0x7fedb40cbc60 2024-03-23T15:52:30.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.365+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb41f0340 con 0x7fedb40cbc60 2024-03-23T15:52:30.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.365+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41f0900 con 0x7fedb40cbc60 2024-03-23T15:52:30.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.366+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb4075740 con 0x7fedb40cbc60 2024-03-23T15:52:30.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.367+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feda4005e40 con 0x7fedb40cbc60 2024-03-23T15:52:30.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.367+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4005230 con 0x7fedb40cbc60 2024-03-23T15:52:30.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.621+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feda4005450 con 0x7fedb40cbc60 2024-03-23T15:52:30.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.621+0000 7fedaaffd640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed9806b810 0x7fed9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:30.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.622+0000 7fedba49e640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed9806b810 0x7fed9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:30.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.622+0000 7fedba49e640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed9806b810 0x7fed9806dcd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009b60 tx=0x7fedb0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.637+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7feda4058350 con 0x7fedb40cbc60 2024-03-23T15:52:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.637+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feda40545f0 con 0x7fedb40cbc60 2024-03-23T15:52:30.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.867+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedb40d0a30 con 0x7fedb40cbc60 2024-03-23T15:52:30.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.925+0000 7fedaaffd640 1 -- 172.21.15.129:0/578518652 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7feda4017090 con 0x7fedb40cbc60 2024-03-23T15:52:30.928 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:30.928 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:30.928 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:30.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed9806b810 msgr2=0x7fed9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:30.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed9806b810 0x7fed9806dcd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009b60 tx=0x7fedb0009190 comp rx=0 tx=0).stop 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cbc60 msgr2=0x7fedb40b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cbc60 0x7fedb40b28f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feda4000f60 tx=0x7feda400c800 comp rx=0 tx=0).stop 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 shutdown_connections 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed9806b810 0x7fed9806dcd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fedb40cc080 0x7fedb40b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedb40cbc60 0x7fedb40b28f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 --2- 172.21.15.129:0/578518652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb406c030 0x7fedb41560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 >> 172.21.15.129:0/578518652 conn(0x7fedb40cfb20 msgr2=0x7fedb40c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 shutdown_connections 2024-03-23T15:52:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:30.927+0000 7fedbc702640 1 -- 172.21.15.129:0/578518652 wait complete. 2024-03-23T15:52:31.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:31.546 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:32.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:32.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.646+0000 7f04852f4640 1 Processor -- start 2024-03-23T15:52:32.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.646+0000 7f04852f4640 1 -- start start 2024-03-23T15:52:32.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.647+0000 7f04852f4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04801544b0 0x7f048015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:32.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.647+0000 7f04852f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04800caa00 0x7f048006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:32.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.647+0000 7f04852f4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04800cbcb0 con 0x7f048006c0b0 2024-03-23T15:52:32.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.647+0000 7f04852f4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f04800c5be0 con 0x7f04801544b0 2024-03-23T15:52:32.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.647+0000 7f04852f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f048015b280 con 0x7f04800caa00 2024-03-23T15:52:32.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.648+0000 7f047effd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04801544b0 0x7f048015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:32.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.648+0000 7f047e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04800caa00 0x7f048006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:32.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.649+0000 7f047e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04800caa00 0x7f048006bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:40944/0 (socket says 172.21.15.129:40944) 2024-03-23T15:52:32.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.649+0000 7f047e7fc640 1 -- 172.21.15.129:0/3536781775 learned_addr learned my addr 172.21.15.129:0/3536781775 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:32.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.651+0000 7f047effd640 1 -- 172.21.15.129:0/3536781775 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04800caa00 msgr2=0x7f048006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:32.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.651+0000 7f047effd640 1 --2- 172.21.15.129:0/3536781775 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04800caa00 0x7f048006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:32.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.651+0000 7f047effd640 1 -- 172.21.15.129:0/3536781775 >> v1:172.21.15.129:6789/0 conn(0x7f048006c0b0 legacy=0x7f04800c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:32.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.651+0000 7f047effd640 1 -- 172.21.15.129:0/3536781775 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048015a9e0 con 0x7f04801544b0 2024-03-23T15:52:32.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.657+0000 7f047effd640 1 --2- 172.21.15.129:0/3536781775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04801544b0 0x7f048015a2e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04700097a0 tx=0x7f047002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc510ee66a121714 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:32.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.660+0000 7f047dffb640 1 -- 172.21.15.129:0/3536781775 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470030610 con 0x7f04801544b0 2024-03-23T15:52:32.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.660+0000 7f047dffb640 1 -- 172.21.15.129:0/3536781775 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0470030770 con 0x7f04801544b0 2024-03-23T15:52:32.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.660+0000 7f047dffb640 1 -- 172.21.15.129:0/3536781775 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470030b00 con 0x7f04801544b0 2024-03-23T15:52:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 -- 172.21.15.129:0/3536781775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04801544b0 msgr2=0x7f048015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 --2- 172.21.15.129:0/3536781775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04801544b0 0x7f048015a2e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04700097a0 tx=0x7f047002b960 comp rx=0 tx=0).stop 2024-03-23T15:52:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 -- 172.21.15.129:0/3536781775 shutdown_connections 2024-03-23T15:52:32.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 --2- 172.21.15.129:0/3536781775 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04800caa00 0x7f048006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:32.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 --2- 172.21.15.129:0/3536781775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04801544b0 0x7f048015a2e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:32.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 -- 172.21.15.129:0/3536781775 >> 172.21.15.129:0/3536781775 conn(0x7f0480061b50 msgr2=0x7f0480061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:32.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 -- 172.21.15.129:0/3536781775 shutdown_connections 2024-03-23T15:52:32.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.661+0000 7f04852f4640 1 -- 172.21.15.129:0/3536781775 wait complete. 2024-03-23T15:52:32.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.663+0000 7f04852f4640 1 Processor -- start 2024-03-23T15:52:32.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.663+0000 7f04852f4640 1 -- start start 2024-03-23T15:52:32.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.664+0000 7f04852f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048006c0b0 0x7f0480148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:32.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.664+0000 7f04852f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 0x7f0480142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:32.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.664+0000 7f04852f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04801544b0 0x7f04801415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:32.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.664+0000 7f04852f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04801563e0 con 0x7f04800caa00 2024-03-23T15:52:32.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.664+0000 7f04852f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04800d0390 con 0x7f04801544b0 2024-03-23T15:52:32.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.664+0000 7f04852f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0480157970 con 0x7f048006c0b0 2024-03-23T15:52:32.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.665+0000 7f047e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 0x7f0480142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:32.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.665+0000 7f047effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048006c0b0 0x7f0480148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:32.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.665+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04801544b0 0x7f04801415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:32.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.665+0000 7f047e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 0x7f0480142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48508/0 (socket says 172.21.15.129:48508) 2024-03-23T15:52:32.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.665+0000 7f047e7fc640 1 -- 172.21.15.129:0/3361096772 learned_addr learned my addr 172.21.15.129:0/3361096772 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.669+0000 7f047e7fc640 1 -- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048006c0b0 msgr2=0x7f0480148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.669+0000 7f047e7fc640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048006c0b0 0x7f0480148540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.669+0000 7f047e7fc640 1 -- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04801544b0 msgr2=0x7f04801415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.669+0000 7f047e7fc640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04801544b0 0x7f04801415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.669+0000 7f047e7fc640 1 -- 172.21.15.129:0/3361096772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0470009400 con 0x7f04800caa00 2024-03-23T15:52:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.672+0000 7f047e7fc640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 0x7f0480142690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0474009fd0 tx=0x7f047400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.676+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047400c940 con 0x7f04800caa00 2024-03-23T15:52:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.676+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f047400caa0 con 0x7f04800caa00 2024-03-23T15:52:32.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.676+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047400ce30 con 0x7f04800caa00 2024-03-23T15:52:32.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.676+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04800d05a0 con 0x7f04800caa00 2024-03-23T15:52:32.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.676+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0480155960 con 0x7f04800caa00 2024-03-23T15:52:32.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.678+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04800ca770 con 0x7f04800caa00 2024-03-23T15:52:32.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.697+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0474013470 con 0x7f04800caa00 2024-03-23T15:52:32.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.697+0000 7f04677fe640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f046c06b860 0x7f046c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:32.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.698+0000 7f047effd640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f046c06b860 0x7f046c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:32.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.698+0000 7f047effd640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f046c06b860 0x7f046c06dd20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0470036000 tx=0x7f047003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:32.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.944+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0474014e40 con 0x7f04800caa00 2024-03-23T15:52:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:32.944+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0474055570 con 0x7f04800caa00 2024-03-23T15:52:33.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.167+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0480075740 con 0x7f04800caa00 2024-03-23T15:52:33.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.234+0000 7f04677fe640 1 -- 172.21.15.129:0/3361096772 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0474017090 con 0x7f04800caa00 2024-03-23T15:52:33.237 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:33.238 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:33.238 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f046c06b860 msgr2=0x7f046c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f046c06b860 0x7f046c06dd20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0470036000 tx=0x7f047003b040 comp rx=0 tx=0).stop 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 msgr2=0x7f0480142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 0x7f0480142690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0474009fd0 tx=0x7f047400ed40 comp rx=0 tx=0).stop 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 shutdown_connections 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f046c06b860 0x7f046c06dd20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04801544b0 0x7f04801415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04800caa00 0x7f0480142690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 --2- 172.21.15.129:0/3361096772 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048006c0b0 0x7f0480148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 >> 172.21.15.129:0/3361096772 conn(0x7f0480061b50 msgr2=0x7f048014a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 shutdown_connections 2024-03-23T15:52:33.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:33.236+0000 7f04852f4640 1 -- 172.21.15.129:0/3361096772 wait complete. 2024-03-23T15:52:33.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:33.858 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:34.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.958+0000 7f5c807e2640 1 Processor -- start 2024-03-23T15:52:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.958+0000 7f5c807e2640 1 -- start start 2024-03-23T15:52:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.959+0000 7f5c807e2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c780c54a0 0x7f5c781581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.959+0000 7f5c807e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c7815c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.959+0000 7f5c807e2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c7815b720 con 0x7f5c780c5880 2024-03-23T15:52:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.959+0000 7f5c807e2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5c780b20a0 con 0x7f5c780c54a0 2024-03-23T15:52:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.959+0000 7f5c807e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5c7806fc10 con 0x7f5c7806f6d0 2024-03-23T15:52:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.960+0000 7f5c7e57e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c780c54a0 0x7f5c781581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.960+0000 7f5c7ed7f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c7815c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.960+0000 7f5c7e57e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c780c54a0 0x7f5c781581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48564/0 (socket says 172.21.15.129:48564) 2024-03-23T15:52:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.960+0000 7f5c7ed7f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c7815c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:40954/0 (socket says 172.21.15.129:40954) 2024-03-23T15:52:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.960+0000 7f5c7e57e640 1 -- 172.21.15.129:0/2654915465 learned_addr learned my addr 172.21.15.129:0/2654915465 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:34.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.963+0000 7f5c7ed7f640 1 -- 172.21.15.129:0/2654915465 >> v2:172.21.15.157:3300/0 conn(0x7f5c780c54a0 msgr2=0x7f5c781581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:34.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.963+0000 7f5c7ed7f640 1 --2- 172.21.15.129:0/2654915465 >> v2:172.21.15.157:3300/0 conn(0x7f5c780c54a0 0x7f5c781581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:34.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.963+0000 7f5c7ed7f640 1 -- 172.21.15.129:0/2654915465 >> v1:172.21.15.157:6789/0 conn(0x7f5c780c5880 legacy=0x7f5c780718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.963+0000 7f5c7ed7f640 1 -- 172.21.15.129:0/2654915465 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c7806fef0 con 0x7f5c7806f6d0 2024-03-23T15:52:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.965+0000 7f5c7ed7f640 1 --2- 172.21.15.129:0/2654915465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c7815c170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009d90 tx=0x7f5c7002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4721ad06cb47a6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:34.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c7d57c640 1 -- 172.21.15.129:0/2654915465 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7002cc50 con 0x7f5c7806f6d0 2024-03-23T15:52:34.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c7d57c640 1 -- 172.21.15.129:0/2654915465 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7002cdb0 con 0x7f5c7806f6d0 2024-03-23T15:52:34.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c7d57c640 1 -- 172.21.15.129:0/2654915465 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c70034660 con 0x7f5c7806f6d0 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c807e2640 1 -- 172.21.15.129:0/2654915465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 msgr2=0x7f5c7815c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c807e2640 1 --2- 172.21.15.129:0/2654915465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c7815c170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009d90 tx=0x7f5c7002bea0 comp rx=0 tx=0).stop 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c807e2640 1 -- 172.21.15.129:0/2654915465 shutdown_connections 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c807e2640 1 --2- 172.21.15.129:0/2654915465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c7815c170 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c807e2640 1 --2- 172.21.15.129:0/2654915465 >> v2:172.21.15.157:3300/0 conn(0x7f5c780c54a0 0x7f5c781581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.970+0000 7f5c807e2640 1 -- 172.21.15.129:0/2654915465 >> 172.21.15.129:0/2654915465 conn(0x7f5c780cbb20 msgr2=0x7f5c780cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.971+0000 7f5c807e2640 1 -- 172.21.15.129:0/2654915465 shutdown_connections 2024-03-23T15:52:34.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.971+0000 7f5c807e2640 1 -- 172.21.15.129:0/2654915465 wait complete. 2024-03-23T15:52:34.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 Processor -- start 2024-03-23T15:52:34.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 -- start start 2024-03-23T15:52:34.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c781f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:34.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c780c54a0 0x7f5c781ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:34.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c780c5880 0x7f5c7815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:34.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c781edec0 con 0x7f5c780c5880 2024-03-23T15:52:34.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5c780cabb0 con 0x7f5c780c54a0 2024-03-23T15:52:34.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.972+0000 7f5c807e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5c781f1e30 con 0x7f5c7806f6d0 2024-03-23T15:52:34.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.973+0000 7f5c7e57e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c781f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:34.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.973+0000 7f5c7dd7d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c780c54a0 0x7f5c781ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:34.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.974+0000 7f5c7dd7d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c780c54a0 0x7f5c781ee2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48566/0 (socket says 172.21.15.129:48566) 2024-03-23T15:52:34.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.974+0000 7f5c7e57e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c781f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:40964/0 (socket says 172.21.15.129:40964) 2024-03-23T15:52:34.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.974+0000 7f5c7dd7d640 1 -- 172.21.15.129:0/91620685 learned_addr learned my addr 172.21.15.129:0/91620685 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:34.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.974+0000 7f5c7ed7f640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c780c5880 0x7f5c7815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.978+0000 7f5c7ed7f640 1 -- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 msgr2=0x7f5c781f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.978+0000 7f5c7ed7f640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c781f18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.978+0000 7f5c7ed7f640 1 -- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c780c54a0 msgr2=0x7f5c781ee2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.978+0000 7f5c7ed7f640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c780c54a0 0x7f5c781ee2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:34.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.978+0000 7f5c7ed7f640 1 -- 172.21.15.129:0/91620685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c70009a70 con 0x7f5c780c5880 2024-03-23T15:52:34.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.980+0000 7f5c7ed7f640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c780c5880 0x7f5c7815a7e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c70033000 tx=0x7f5c70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.985+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c70040650 con 0x7f5c780c5880 2024-03-23T15:52:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.985+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c700407b0 con 0x7f5c780c5880 2024-03-23T15:52:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.985+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c780cae50 con 0x7f5c780c5880 2024-03-23T15:52:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.985+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c70040b60 con 0x7f5c780c5880 2024-03-23T15:52:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.985+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c780cb410 con 0x7f5c780c5880 2024-03-23T15:52:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:34.986+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c7814e2b0 con 0x7f5c780c5880 2024-03-23T15:52:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.015+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5c70040cc0 con 0x7f5c780c5880 2024-03-23T15:52:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.015+0000 7f5c6effd640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5c5c06b810 0x7f5c5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:35.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.015+0000 7f5c7e57e640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5c5c06b810 0x7f5c5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:35.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.016+0000 7f5c7e57e640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5c5c06b810 0x7f5c5c06dcd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009b30 tx=0x7f5c74009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:35.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.021+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5c7007bae0 con 0x7f5c780c5880 2024-03-23T15:52:35.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.276+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5c70077d80 con 0x7f5c780c5880 2024-03-23T15:52:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.514+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c781ee000 con 0x7f5c780c5880 2024-03-23T15:52:35.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.535+0000 7f5c6effd640 1 -- 172.21.15.129:0/91620685 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5c781ee000 con 0x7f5c780c5880 2024-03-23T15:52:35.538 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:35.538 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:35.538 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5c5c06b810 msgr2=0x7f5c5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5c5c06b810 0x7f5c5c06dcd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009b30 tx=0x7f5c74009240 comp rx=0 tx=0).stop 2024-03-23T15:52:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c780c5880 msgr2=0x7f5c7815a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c780c5880 0x7f5c7815a7e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c70033000 tx=0x7f5c70004a40 comp rx=0 tx=0).stop 2024-03-23T15:52:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 shutdown_connections 2024-03-23T15:52:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c780c5880 0x7f5c7815a7e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5c5c06b810 0x7f5c5c06dcd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c780c54a0 0x7f5c781ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 --2- 172.21.15.129:0/91620685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c7806f6d0 0x7f5c781f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 >> 172.21.15.129:0/91620685 conn(0x7f5c780cbb20 msgr2=0x7f5c780718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 shutdown_connections 2024-03-23T15:52:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:35.537+0000 7f5c807e2640 1 -- 172.21.15.129:0/91620685 wait complete. 2024-03-23T15:52:36.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:36.157 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:37.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:52:37.125 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:52:37.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:37.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.264+0000 7fb6a5215640 1 Processor -- start 2024-03-23T15:52:37.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.264+0000 7fb6a5215640 1 -- start start 2024-03-23T15:52:37.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.265+0000 7fb6a5215640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a006c030 0x7fb6a006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.265+0000 7fb6a5215640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb6a00cbc60 0x7fb6a015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.265+0000 7fb6a5215640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb6a00cc080 0x7fb6a015de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.265+0000 7fb6a5215640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a006cae0 con 0x7fb6a00cc080 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.265+0000 7fb6a5215640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a00cabc0 con 0x7fb6a00cbc60 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.265+0000 7fb6a5215640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a00caea0 con 0x7fb6a006c030 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.266+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a006c030 0x7fb6a006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.266+0000 7fb69e59c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb6a00cbc60 0x7fb6a015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.267+0000 7fb69e59c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb6a00cbc60 0x7fb6a015d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:48570/0 (socket says 172.21.15.129:48570) 2024-03-23T15:52:37.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.267+0000 7fb69e59c640 1 -- 172.21.15.129:0/641068976 learned_addr learned my addr 172.21.15.129:0/641068976 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:37.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.268+0000 7fb69f59e640 1 --2- 172.21.15.129:0/641068976 >> v2:172.21.15.129:3300/0 conn(0x7fb6a00cc080 0x7fb6a015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.269+0000 7fb69ed9d640 1 -- 172.21.15.129:0/641068976 >> v2:172.21.15.157:3300/0 conn(0x7fb6a00cbc60 msgr2=0x7fb6a015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:37.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.269+0000 7fb69ed9d640 1 --2- 172.21.15.129:0/641068976 >> v2:172.21.15.157:3300/0 conn(0x7fb6a00cbc60 0x7fb6a015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.269+0000 7fb69ed9d640 1 -- 172.21.15.129:0/641068976 >> v2:172.21.15.129:3300/0 conn(0x7fb6a00cc080 msgr2=0x7fb6a015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:37.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.269+0000 7fb69ed9d640 1 --2- 172.21.15.129:0/641068976 >> v2:172.21.15.129:3300/0 conn(0x7fb6a00cc080 0x7fb6a015de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.269+0000 7fb69ed9d640 1 -- 172.21.15.129:0/641068976 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a00cb180 con 0x7fb6a006c030 2024-03-23T15:52:37.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.269+0000 7fb69f59e640 1 --2- 172.21.15.129:0/641068976 >> v2:172.21.15.129:3300/0 conn(0x7fb6a00cc080 0x7fb6a015de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T15:52:37.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.271+0000 7fb69ed9d640 1 --2- 172.21.15.129:0/641068976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a006c030 0x7fb6a006c410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb6a008cc70 tx=0x7fb69002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795ea5fb044d304b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:37.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.275+0000 7fb69dd9b640 1 -- 172.21.15.129:0/641068976 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69002f050 con 0x7fb6a006c030 2024-03-23T15:52:37.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.275+0000 7fb69dd9b640 1 -- 172.21.15.129:0/641068976 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb690031700 con 0x7fb6a006c030 2024-03-23T15:52:37.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.275+0000 7fb69dd9b640 1 -- 172.21.15.129:0/641068976 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690031a90 con 0x7fb6a006c030 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 -- 172.21.15.129:0/641068976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a006c030 msgr2=0x7fb6a006c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 --2- 172.21.15.129:0/641068976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a006c030 0x7fb6a006c410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb6a008cc70 tx=0x7fb69002b7a0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 -- 172.21.15.129:0/641068976 shutdown_connections 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 --2- 172.21.15.129:0/641068976 >> v2:172.21.15.129:3300/0 conn(0x7fb6a00cc080 0x7fb6a015de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 --2- 172.21.15.129:0/641068976 >> v2:172.21.15.157:3300/0 conn(0x7fb6a00cbc60 0x7fb6a015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 --2- 172.21.15.129:0/641068976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a006c030 0x7fb6a006c410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 -- 172.21.15.129:0/641068976 >> 172.21.15.129:0/641068976 conn(0x7fb6a00cfb20 msgr2=0x7fb6a00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 -- 172.21.15.129:0/641068976 shutdown_connections 2024-03-23T15:52:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.276+0000 7fb6a5215640 1 -- 172.21.15.129:0/641068976 wait complete. 2024-03-23T15:52:37.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.278+0000 7fb6a5215640 1 Processor -- start 2024-03-23T15:52:37.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.278+0000 7fb6a5215640 1 -- start start 2024-03-23T15:52:37.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.279+0000 7fb6a5215640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 0x7fb6a015b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.279+0000 7fb6a5215640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a00cbc60 0x7fb6a01efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.279+0000 7fb6a5215640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a00cc080 0x7fb6a01f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.279+0000 7fb6a5215640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a015c910 con 0x7fb6a006c030 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.279+0000 7fb6a5215640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a015d130 con 0x7fb6a00cc080 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.279+0000 7fb6a5215640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a015c270 con 0x7fb6a00cbc60 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.280+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 0x7fb6a015b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.280+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a00cbc60 0x7fb6a01efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.280+0000 7fb69f59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a00cc080 0x7fb6a01f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.280+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 0x7fb6a015b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48530/0 (socket says 172.21.15.129:48530) 2024-03-23T15:52:37.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.280+0000 7fb69ed9d640 1 -- 172.21.15.129:0/3443848243 learned_addr learned my addr 172.21.15.129:0/3443848243 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.285+0000 7fb69ed9d640 1 -- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a00cbc60 msgr2=0x7fb6a01efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.285+0000 7fb69ed9d640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a00cbc60 0x7fb6a01efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.285+0000 7fb69ed9d640 1 -- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a00cc080 msgr2=0x7fb6a01f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.285+0000 7fb69ed9d640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a00cc080 0x7fb6a01f00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.285+0000 7fb69ed9d640 1 -- 172.21.15.129:0/3443848243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb690009400 con 0x7fb6a006c030 2024-03-23T15:52:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.285+0000 7fb69e59c640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a00cbc60 0x7fb6a01efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:37.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.288+0000 7fb69ed9d640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 0x7fb6a015b760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb6900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:37.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.291+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900339b0 con 0x7fb6a006c030 2024-03-23T15:52:37.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.291+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a015e350 con 0x7fb6a006c030 2024-03-23T15:52:37.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.291+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a015c530 con 0x7fb6a006c030 2024-03-23T15:52:37.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.292+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a00cb700 con 0x7fb6a006c030 2024-03-23T15:52:37.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.503+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb690033b10 con 0x7fb6a006c030 2024-03-23T15:52:37.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.592+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69002f020 con 0x7fb6a006c030 2024-03-23T15:52:37.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.592+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb69003a040 con 0x7fb6a006c030 2024-03-23T15:52:37.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.593+0000 7fb6877fe640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb68c06b810 0x7fb68c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:37.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.593+0000 7fb69e59c640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb68c06b810 0x7fb68c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:37.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.593+0000 7fb69e59c640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb68c06b810 0x7fb68c06dcd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb6a01545f0 tx=0x7fb694009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:37.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.614+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb69007ab10 con 0x7fb6a006c030 2024-03-23T15:52:37.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.614+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb690076db0 con 0x7fb6a006c030 2024-03-23T15:52:37.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.900+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6a015c6c0 con 0x7fb6a006c030 2024-03-23T15:52:37.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.914+0000 7fb6877fe640 1 -- 172.21.15.129:0/3443848243 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb690041090 con 0x7fb6a006c030 2024-03-23T15:52:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:37.918 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:37.918 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:37.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb68c06b810 msgr2=0x7fb68c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:37.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb68c06b810 0x7fb68c06dcd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb6a01545f0 tx=0x7fb694009240 comp rx=0 tx=0).stop 2024-03-23T15:52:37.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 msgr2=0x7fb6a015b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:37.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 0x7fb6a015b760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb6900046c0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 shutdown_connections 2024-03-23T15:52:37.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb68c06b810 0x7fb68c06dcd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a00cc080 0x7fb6a01f00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6a00cbc60 0x7fb6a01efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 --2- 172.21.15.129:0/3443848243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a006c030 0x7fb6a015b760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:37.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 >> 172.21.15.129:0/3443848243 conn(0x7fb6a00cfb20 msgr2=0x7fb6a00c8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:37.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 shutdown_connections 2024-03-23T15:52:37.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:37.917+0000 7fb6a5215640 1 -- 172.21.15.129:0/3443848243 wait complete. 2024-03-23T15:52:38.539 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:38.539 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:39.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:39.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.639+0000 7fd645d5c640 1 Processor -- start 2024-03-23T15:52:39.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.640+0000 7fd645d5c640 1 -- start start 2024-03-23T15:52:39.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.640+0000 7fd645d5c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd64006cf70 0x7fd6400cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:39.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.640+0000 7fd645d5c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd640070640 con 0x7fd64015df80 2024-03-23T15:52:39.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.640+0000 7fd645d5c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6400c4c00 con 0x7fd64015dba0 2024-03-23T15:52:39.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.640+0000 7fd645d5c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd64006c240 con 0x7fd64006cf70 2024-03-23T15:52:39.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.641+0000 7fd63effd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd64006cf70 0x7fd6400cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:39.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.641+0000 7fd63f7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd64015dba0 0x7fd64006ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:41020/0 (socket says 172.21.15.129:41020) 2024-03-23T15:52:39.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.641+0000 7fd63f7fe640 1 -- 172.21.15.129:0/80769130 learned_addr learned my addr 172.21.15.129:0/80769130 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.644+0000 7fd63e7fc640 1 -- 172.21.15.129:0/80769130 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661134009 0 0) 0x7fd630001110 con 0x7fd64015dba0 2024-03-23T15:52:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.644+0000 7fd63e7fc640 1 -- 172.21.15.129:0/80769130 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd628003520 con 0x7fd64015dba0 2024-03-23T15:52:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.644+0000 7fd63effd640 1 -- 172.21.15.129:0/80769130 >> v1:172.21.15.157:6789/0 conn(0x7fd64015dba0 legacy=0x7fd64006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.644+0000 7fd63effd640 1 -- 172.21.15.129:0/80769130 >> v1:172.21.15.129:6789/0 conn(0x7fd64015df80 legacy=0x7fd6401540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.644+0000 7fd63effd640 1 -- 172.21.15.129:0/80769130 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6400cbb20 con 0x7fd64006cf70 2024-03-23T15:52:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.650+0000 7fd63effd640 1 --2- 172.21.15.129:0/80769130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64006cf70 0x7fd6400cca30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd6340097c0 tx=0x7fd63402b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa65bb307052792 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.654+0000 7fd63e7fc640 1 -- 172.21.15.129:0/80769130 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634030880 con 0x7fd64006cf70 2024-03-23T15:52:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.654+0000 7fd63e7fc640 1 -- 172.21.15.129:0/80769130 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6340309e0 con 0x7fd64006cf70 2024-03-23T15:52:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.654+0000 7fd63e7fc640 1 -- 172.21.15.129:0/80769130 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634030d70 con 0x7fd64006cf70 2024-03-23T15:52:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 -- 172.21.15.129:0/80769130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64006cf70 msgr2=0x7fd6400cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 --2- 172.21.15.129:0/80769130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64006cf70 0x7fd6400cca30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd6340097c0 tx=0x7fd63402b9c0 comp rx=0 tx=0).stop 2024-03-23T15:52:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 -- 172.21.15.129:0/80769130 shutdown_connections 2024-03-23T15:52:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 --2- 172.21.15.129:0/80769130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64006cf70 0x7fd6400cca30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 -- 172.21.15.129:0/80769130 >> 172.21.15.129:0/80769130 conn(0x7fd640071300 msgr2=0x7fd6400716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 -- 172.21.15.129:0/80769130 shutdown_connections 2024-03-23T15:52:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.655+0000 7fd645d5c640 1 -- 172.21.15.129:0/80769130 wait complete. 2024-03-23T15:52:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.656+0000 7fd645d5c640 1 Processor -- start 2024-03-23T15:52:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 -- start start 2024-03-23T15:52:39.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006cf70 0x7fd640157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:39.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd64015dba0 0x7fd6401e3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:39.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 0x7fd640156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd640155810 con 0x7fd64015df80 2024-03-23T15:52:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd6401e58e0 con 0x7fd64015dba0 2024-03-23T15:52:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.657+0000 7fd645d5c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6401e4e10 con 0x7fd64006cf70 2024-03-23T15:52:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.658+0000 7fd63ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 0x7fd640156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.658+0000 7fd63f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006cf70 0x7fd640157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.658+0000 7fd63ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 0x7fd640156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48568/0 (socket says 172.21.15.129:48568) 2024-03-23T15:52:39.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.658+0000 7fd63ffff640 1 -- 172.21.15.129:0/1495968525 learned_addr learned my addr 172.21.15.129:0/1495968525 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:39.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.658+0000 7fd63effd640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd64015dba0 0x7fd6401e3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:39.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.661+0000 7fd63ffff640 1 -- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006cf70 msgr2=0x7fd640157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:39.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.661+0000 7fd63ffff640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006cf70 0x7fd640157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:39.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.661+0000 7fd63ffff640 1 -- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd64015dba0 msgr2=0x7fd6401e3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:39.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.661+0000 7fd63ffff640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd64015dba0 0x7fd6401e3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:39.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.661+0000 7fd63ffff640 1 -- 172.21.15.129:0/1495968525 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6340093a0 con 0x7fd64015df80 2024-03-23T15:52:39.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.664+0000 7fd63ffff640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 0x7fd640156200 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c6b0 tx=0x7fd63800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:39.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.667+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638003b70 con 0x7fd64015df80 2024-03-23T15:52:39.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.667+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd638003cf0 con 0x7fd64015df80 2024-03-23T15:52:39.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.667+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6401e4b50 con 0x7fd64015df80 2024-03-23T15:52:39.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.667+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638003ea0 con 0x7fd64015df80 2024-03-23T15:52:39.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.667+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6401e5320 con 0x7fd64015df80 2024-03-23T15:52:39.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.668+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6400686f0 con 0x7fd64015df80 2024-03-23T15:52:39.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.687+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd638005310 con 0x7fd64015df80 2024-03-23T15:52:39.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.687+0000 7fd627fff640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62c06b860 0x7fd62c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:39.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.687+0000 7fd63f7fe640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62c06b860 0x7fd62c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:39.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.688+0000 7fd63f7fe640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62c06b860 0x7fd62c06dd20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd630001270 tx=0x7fd630007bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:39.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.936+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd638058f60 con 0x7fd64015df80 2024-03-23T15:52:39.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:39.936+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd638055200 con 0x7fd64015df80 2024-03-23T15:52:40.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.176+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6401e4b50 con 0x7fd64015df80 2024-03-23T15:52:40.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.188+0000 7fd627fff640 1 -- 172.21.15.129:0/1495968525 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd6401e4b50 con 0x7fd64015df80 2024-03-23T15:52:40.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:40.191 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:40.192 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62c06b860 msgr2=0x7fd62c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62c06b860 0x7fd62c06dd20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd630001270 tx=0x7fd630007bb0 comp rx=0 tx=0).stop 2024-03-23T15:52:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 msgr2=0x7fd640156200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:40.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 0x7fd640156200 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c6b0 tx=0x7fd63800cb70 comp rx=0 tx=0).stop 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 shutdown_connections 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd64015df80 0x7fd640156200 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd62c06b860 0x7fd62c06dd20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd64015dba0 0x7fd6401e3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 --2- 172.21.15.129:0/1495968525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64006cf70 0x7fd640157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 >> 172.21.15.129:0/1495968525 conn(0x7fd640071300 msgr2=0x7fd6400c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 shutdown_connections 2024-03-23T15:52:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:40.191+0000 7fd645d5c640 1 -- 172.21.15.129:0/1495968525 wait complete. 2024-03-23T15:52:40.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:40.813 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:41.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:41.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.915+0000 7f6c8067c640 1 Processor -- start 2024-03-23T15:52:41.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.915+0000 7f6c8067c640 1 -- start start 2024-03-23T15:52:41.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.916+0000 7f6c8067c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6c7806c410 0x7f6c7815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:41.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.916+0000 7f6c8067c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c7806c830 con 0x7f6c7806c030 2024-03-23T15:52:41.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.916+0000 7f6c8067c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c780cc370 con 0x7f6c7815de40 2024-03-23T15:52:41.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.916+0000 7f6c8067c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6c7806cae0 con 0x7f6c7806c410 2024-03-23T15:52:41.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.916+0000 7f6c7dc17640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6c7806c410 0x7f6c7815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:41.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.917+0000 7f6c7ec19640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6c7815de40 0x7f6c78164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:37154/0 (socket says 172.21.15.129:37154) 2024-03-23T15:52:41.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.917+0000 7f6c7ec19640 1 -- 172.21.15.129:0/3394922073 learned_addr learned my addr 172.21.15.129:0/3394922073 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:41.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.920+0000 7f6c7dc17640 1 -- 172.21.15.129:0/3394922073 >> v1:172.21.15.157:6789/0 conn(0x7f6c7815de40 legacy=0x7f6c78164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:41.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.920+0000 7f6c7dc17640 1 -- 172.21.15.129:0/3394922073 >> v1:172.21.15.129:6789/0 conn(0x7f6c7806c030 legacy=0x7f6c780cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:41.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.920+0000 7f6c7dc17640 1 -- 172.21.15.129:0/3394922073 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c780cabc0 con 0x7f6c7806c410 2024-03-23T15:52:41.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.925+0000 7f6c7dc17640 1 --2- 172.21.15.129:0/3394922073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c7815d900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6c680097a0 tx=0x7f6c6802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec4a32f045bad94 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:41.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.927+0000 7f6c7d416640 1 -- 172.21.15.129:0/3394922073 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68030610 con 0x7f6c7806c410 2024-03-23T15:52:41.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.929+0000 7f6c7d416640 1 -- 172.21.15.129:0/3394922073 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c68030770 con 0x7f6c7806c410 2024-03-23T15:52:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c7d416640 1 -- 172.21.15.129:0/3394922073 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68030b00 con 0x7f6c7806c410 2024-03-23T15:52:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 -- 172.21.15.129:0/3394922073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 msgr2=0x7f6c7815d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 --2- 172.21.15.129:0/3394922073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c7815d900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6c680097a0 tx=0x7f6c6802b960 comp rx=0 tx=0).stop 2024-03-23T15:52:41.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 -- 172.21.15.129:0/3394922073 shutdown_connections 2024-03-23T15:52:41.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 --2- 172.21.15.129:0/3394922073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c7815d900 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:41.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 -- 172.21.15.129:0/3394922073 >> 172.21.15.129:0/3394922073 conn(0x7f6c780cfb20 msgr2=0x7f6c780cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:41.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 -- 172.21.15.129:0/3394922073 shutdown_connections 2024-03-23T15:52:41.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.930+0000 7f6c8067c640 1 -- 172.21.15.129:0/3394922073 wait complete. 2024-03-23T15:52:41.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.932+0000 7f6c8067c640 1 Processor -- start 2024-03-23T15:52:41.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.932+0000 7f6c8067c640 1 -- start start 2024-03-23T15:52:41.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.932+0000 7f6c8067c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c7806c030 0x7f6c78154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:41.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c8067c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c78153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:41.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c8067c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c7815de40 0x7f6c78154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:41.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c8067c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c780ca230 con 0x7f6c7806c410 2024-03-23T15:52:41.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c8067c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781f4820 con 0x7f6c7806c030 2024-03-23T15:52:41.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c8067c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6c78156d10 con 0x7f6c7815de40 2024-03-23T15:52:41.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c7dc17640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c78153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:41.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c7e418640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c7806c030 0x7f6c78154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:41.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c7ec19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c7815de40 0x7f6c78154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:41.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c7dc17640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c78153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39872/0 (socket says 172.21.15.129:39872) 2024-03-23T15:52:41.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.933+0000 7f6c7dc17640 1 -- 172.21.15.129:0/515700795 learned_addr learned my addr 172.21.15.129:0/515700795 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:41.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.937+0000 7f6c7dc17640 1 -- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c7815de40 msgr2=0x7f6c78154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:41.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.937+0000 7f6c7dc17640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c7815de40 0x7f6c78154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:41.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.937+0000 7f6c7dc17640 1 -- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c7806c030 msgr2=0x7f6c78154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:41.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.937+0000 7f6c7dc17640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c7806c030 0x7f6c78154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:41.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.937+0000 7f6c7dc17640 1 -- 172.21.15.129:0/515700795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c68009400 con 0x7f6c7806c410 2024-03-23T15:52:41.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.939+0000 7f6c7dc17640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c78153e50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6c6802bf70 tx=0x7f6c68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:41.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.941+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6803e040 con 0x7f6c7806c410 2024-03-23T15:52:41.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.941+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c78156ff0 con 0x7f6c7806c410 2024-03-23T15:52:41.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.941+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c781575b0 con 0x7f6c7806c410 2024-03-23T15:52:41.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.942+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c78157740 con 0x7f6c7806c410 2024-03-23T15:52:41.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.944+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c68039c90 con 0x7f6c7806c410 2024-03-23T15:52:41.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:41.944+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6804e440 con 0x7f6c7806c410 2024-03-23T15:52:42.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.217+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6c680443f0 con 0x7f6c7806c410 2024-03-23T15:52:42.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.218+0000 7f6c6effd640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6c5c06b840 0x7f6c5c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:42.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.218+0000 7f6c7e418640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6c5c06b840 0x7f6c5c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:42.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.219+0000 7f6c7e418640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6c5c06b840 0x7f6c5c06dd00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c740016e0 tx=0x7f6c74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.231+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6c6807b450 con 0x7f6c7806c410 2024-03-23T15:52:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.231+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6c680776f0 con 0x7f6c7806c410 2024-03-23T15:52:42.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.463+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c78075740 con 0x7f6c7806c410 2024-03-23T15:52:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.486+0000 7f6c6effd640 1 -- 172.21.15.129:0/515700795 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6c68034090 con 0x7f6c7806c410 2024-03-23T15:52:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:42.490 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6c5c06b840 msgr2=0x7f6c5c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6c5c06b840 0x7f6c5c06dd00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c740016e0 tx=0x7f6c74009040 comp rx=0 tx=0).stop 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 msgr2=0x7f6c78153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c78153e50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6c6802bf70 tx=0x7f6c68004730 comp rx=0 tx=0).stop 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 shutdown_connections 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c7815de40 0x7f6c78154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7806c410 0x7f6c78153e50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6c5c06b840 0x7f6c5c06dd00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 --2- 172.21.15.129:0/515700795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c7806c030 0x7f6c78154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 >> 172.21.15.129:0/515700795 conn(0x7f6c780cfb20 msgr2=0x7f6c780c7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 shutdown_connections 2024-03-23T15:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:42.489+0000 7f6c8067c640 1 -- 172.21.15.129:0/515700795 wait complete. 2024-03-23T15:52:43.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:43.109 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:44.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:44.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.209+0000 7fca160c3640 1 Processor -- start 2024-03-23T15:52:44.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.209+0000 7fca160c3640 1 -- start start 2024-03-23T15:52:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.210+0000 7fca160c3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fca100ccd60 0x7fca1015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.210+0000 7fca160c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ceb00 0x7fca1006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.210+0000 7fca160c3640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fca100cef40 0x7fca1006c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.210+0000 7fca160c3640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fca1015b290 con 0x7fca100cef40 2024-03-23T15:52:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.210+0000 7fca160c3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fca100ce800 con 0x7fca100ccd60 2024-03-23T15:52:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.210+0000 7fca160c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fca10154a70 con 0x7fca100ceb00 2024-03-23T15:52:44.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.211+0000 7fca0f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fca100ccd60 0x7fca1015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.211+0000 7fca0effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ceb00 0x7fca1006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.211+0000 7fca0ffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fca100cef40 0x7fca1006c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.211+0000 7fca0f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fca100ccd60 0x7fca1015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40032/0 (socket says 172.21.15.129:40032) 2024-03-23T15:52:44.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.211+0000 7fca0f7fe640 1 -- 172.21.15.129:0/399561038 learned_addr learned my addr 172.21.15.129:0/399561038 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:44.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.213+0000 7fca0ffff640 1 -- 172.21.15.129:0/399561038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ceb00 msgr2=0x7fca1006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:44.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.213+0000 7fca0ffff640 1 --2- 172.21.15.129:0/399561038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ceb00 0x7fca1006c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.213+0000 7fca0ffff640 1 -- 172.21.15.129:0/399561038 >> v2:172.21.15.157:3300/0 conn(0x7fca100ccd60 msgr2=0x7fca1015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:44.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.213+0000 7fca0ffff640 1 --2- 172.21.15.129:0/399561038 >> v2:172.21.15.157:3300/0 conn(0x7fca100ccd60 0x7fca1015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.213+0000 7fca0ffff640 1 -- 172.21.15.129:0/399561038 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca100cd950 con 0x7fca100cef40 2024-03-23T15:52:44.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.215+0000 7fca0ffff640 1 --2- 172.21.15.129:0/399561038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100cef40 0x7fca1006c7b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fca1008cf40 tx=0x7fca0802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb434d7d9238529 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.218+0000 7fca0e7fc640 1 -- 172.21.15.129:0/399561038 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0802cc60 con 0x7fca100cef40 2024-03-23T15:52:44.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.220+0000 7fca0e7fc640 1 -- 172.21.15.129:0/399561038 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0802cdc0 con 0x7fca100cef40 2024-03-23T15:52:44.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.220+0000 7fca0e7fc640 1 -- 172.21.15.129:0/399561038 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08034660 con 0x7fca100cef40 2024-03-23T15:52:44.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 -- 172.21.15.129:0/399561038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100cef40 msgr2=0x7fca1006c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:44.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 --2- 172.21.15.129:0/399561038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100cef40 0x7fca1006c7b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fca1008cf40 tx=0x7fca0802beb0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 -- 172.21.15.129:0/399561038 shutdown_connections 2024-03-23T15:52:44.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 --2- 172.21.15.129:0/399561038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100cef40 0x7fca1006c7b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 --2- 172.21.15.129:0/399561038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ceb00 0x7fca1006c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 --2- 172.21.15.129:0/399561038 >> v2:172.21.15.157:3300/0 conn(0x7fca100ccd60 0x7fca1015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 -- 172.21.15.129:0/399561038 >> 172.21.15.129:0/399561038 conn(0x7fca100b1840 msgr2=0x7fca100b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 -- 172.21.15.129:0/399561038 shutdown_connections 2024-03-23T15:52:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.221+0000 7fca160c3640 1 -- 172.21.15.129:0/399561038 wait complete. 2024-03-23T15:52:44.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.223+0000 7fca160c3640 1 Processor -- start 2024-03-23T15:52:44.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.223+0000 7fca160c3640 1 -- start start 2024-03-23T15:52:44.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca160c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ccd60 0x7fca101f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca160c3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 0x7fca101527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca160c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca100cef40 0x7fca100ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca160c3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca1014ff50 con 0x7fca100ceb00 2024-03-23T15:52:44.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca160c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca101f4c90 con 0x7fca100cef40 2024-03-23T15:52:44.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca160c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fca101f1430 con 0x7fca100ccd60 2024-03-23T15:52:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 0x7fca101527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca0ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca100cef40 0x7fca100ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.224+0000 7fca0f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ccd60 0x7fca101f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.225+0000 7fca0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 0x7fca101527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39894/0 (socket says 172.21.15.129:39894) 2024-03-23T15:52:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.225+0000 7fca0ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca100cef40 0x7fca100ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40040/0 (socket says 172.21.15.129:40040) 2024-03-23T15:52:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.225+0000 7fca0effd640 1 -- 172.21.15.129:0/1262196293 learned_addr learned my addr 172.21.15.129:0/1262196293 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:44.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.228+0000 7fca0effd640 1 -- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ccd60 msgr2=0x7fca101f4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:44.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.228+0000 7fca0effd640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ccd60 0x7fca101f4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.228+0000 7fca0effd640 1 -- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca100cef40 msgr2=0x7fca100ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:44.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.228+0000 7fca0effd640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca100cef40 0x7fca100ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.228+0000 7fca0effd640 1 -- 172.21.15.129:0/1262196293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca08009a70 con 0x7fca100ceb00 2024-03-23T15:52:44.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.230+0000 7fca0effd640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 0x7fca101527c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fca0400e870 tx=0x7fca0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:44.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.232+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca040094f0 con 0x7fca100ceb00 2024-03-23T15:52:44.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.232+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca100cbff0 con 0x7fca100ceb00 2024-03-23T15:52:44.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.232+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca101f0de0 con 0x7fca100ceb00 2024-03-23T15:52:44.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.234+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1004f530 con 0x7fca100ceb00 2024-03-23T15:52:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.438+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca04005990 con 0x7fca100ceb00 2024-03-23T15:52:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.566+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04005d00 con 0x7fca100ceb00 2024-03-23T15:52:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.567+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fca0400c940 con 0x7fca100ceb00 2024-03-23T15:52:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.567+0000 7fc9f7fff640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9fc06b840 0x7fc9fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.568+0000 7fca0f7fe640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9fc06b840 0x7fc9fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.568+0000 7fca0f7fe640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9fc06b840 0x7fc9fc06dd00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca00009b30 tx=0x7fca00009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:44.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.580+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fca04059190 con 0x7fca100ceb00 2024-03-23T15:52:44.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.580+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fca04055430 con 0x7fca100ceb00 2024-03-23T15:52:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.885+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca1006db50 con 0x7fca100ceb00 2024-03-23T15:52:44.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.899+0000 7fc9f7fff640 1 -- 172.21.15.129:0/1262196293 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fca04021020 con 0x7fca100ceb00 2024-03-23T15:52:44.902 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:44.902 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9fc06b840 msgr2=0x7fc9fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9fc06b840 0x7fc9fc06dd00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca00009b30 tx=0x7fca00009240 comp rx=0 tx=0).stop 2024-03-23T15:52:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 msgr2=0x7fca101527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 0x7fca101527c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fca0400e870 tx=0x7fca0400ed30 comp rx=0 tx=0).stop 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 shutdown_connections 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9fc06b840 0x7fc9fc06dd00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca100cef40 0x7fca100ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca100ceb00 0x7fca101527c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 --2- 172.21.15.129:0/1262196293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca100ccd60 0x7fca101f4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 >> 172.21.15.129:0/1262196293 conn(0x7fca100b1840 msgr2=0x7fca1014d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 shutdown_connections 2024-03-23T15:52:44.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:44.901+0000 7fca160c3640 1 -- 172.21.15.129:0/1262196293 wait complete. 2024-03-23T15:52:45.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:45.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:46.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:46.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.625+0000 7f042b6af640 1 Processor -- start 2024-03-23T15:52:46.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.625+0000 7f042b6af640 1 -- start start 2024-03-23T15:52:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f042b6af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04240ceb00 0x7f042415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f042b6af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f042415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f042b6af640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f042415b290 con 0x7f04240ccd60 2024-03-23T15:52:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f042b6af640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f04240ce800 con 0x7f04240ceb00 2024-03-23T15:52:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f042b6af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0424154a70 con 0x7f04240cef40 2024-03-23T15:52:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f0429c4c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f042415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:46.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.626+0000 7f0428c4a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04240ceb00 0x7f042415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:46.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.627+0000 7f0428c4a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04240ceb00 0x7f042415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40054/0 (socket says 172.21.15.129:40054) 2024-03-23T15:52:46.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.627+0000 7f0428c4a640 1 -- 172.21.15.129:0/2525680933 learned_addr learned my addr 172.21.15.129:0/2525680933 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.629+0000 7f0428c4a640 1 -- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 msgr2=0x7f042415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.629+0000 7f0428c4a640 1 --2- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f042415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.629+0000 7f0428c4a640 1 -- 172.21.15.129:0/2525680933 >> v1:172.21.15.129:6789/0 conn(0x7f04240ccd60 legacy=0x7f042406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.629+0000 7f0428c4a640 1 -- 172.21.15.129:0/2525680933 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04240cd950 con 0x7f04240ceb00 2024-03-23T15:52:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.633+0000 7f0428c4a640 1 --2- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ceb00 0x7f042415ad50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0414009870 tx=0x7f041402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e417b3b40fe78fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:46.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.638+0000 7f041bfff640 1 -- 172.21.15.129:0/2525680933 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414030840 con 0x7f04240ceb00 2024-03-23T15:52:46.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.638+0000 7f041bfff640 1 -- 172.21.15.129:0/2525680933 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04140309a0 con 0x7f04240ceb00 2024-03-23T15:52:46.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f041bfff640 1 -- 172.21.15.129:0/2525680933 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414030d30 con 0x7f04240ceb00 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 -- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ceb00 msgr2=0x7f042415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 --2- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ceb00 0x7f042415ad50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0414009870 tx=0x7f041402b980 comp rx=0 tx=0).stop 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 -- 172.21.15.129:0/2525680933 shutdown_connections 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 --2- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f042415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 --2- 172.21.15.129:0/2525680933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ceb00 0x7f042415ad50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 -- 172.21.15.129:0/2525680933 >> 172.21.15.129:0/2525680933 conn(0x7f04240b1840 msgr2=0x7f04240b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 -- 172.21.15.129:0/2525680933 shutdown_connections 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.639+0000 7f042b6af640 1 -- 172.21.15.129:0/2525680933 wait complete. 2024-03-23T15:52:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 Processor -- start 2024-03-23T15:52:46.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 -- start start 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ccd60 0x7f04241f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04240ceb00 0x7f0424159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f04240cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04240cad20 con 0x7f04240ceb00 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04240cc0b0 con 0x7f04240ccd60 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.641+0000 7f042b6af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04241f0fa0 con 0x7f04240cef40 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.642+0000 7f0428c4a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04240ceb00 0x7f0424159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.642+0000 7f0429c4c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f04240cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:46.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.642+0000 7f042944b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ccd60 0x7f04241f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:46.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.642+0000 7f0428c4a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04240ceb00 0x7f0424159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39908/0 (socket says 172.21.15.129:39908) 2024-03-23T15:52:46.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.642+0000 7f0428c4a640 1 -- 172.21.15.129:0/1781678246 learned_addr learned my addr 172.21.15.129:0/1781678246 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:46.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.647+0000 7f042944b640 1 -- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 msgr2=0x7f04240cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:46.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.647+0000 7f042944b640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f04240cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:46.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.647+0000 7f042944b640 1 -- 172.21.15.129:0/1781678246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04240ceb00 msgr2=0x7f0424159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:46.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.647+0000 7f042944b640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04240ceb00 0x7f0424159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:46.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.647+0000 7f042944b640 1 -- 172.21.15.129:0/1781678246 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04140094d0 con 0x7f04240ccd60 2024-03-23T15:52:46.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.647+0000 7f0429c4c640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f04240cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:46.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.651+0000 7f042944b640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ccd60 0x7f04241f0a60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f042000c410 tx=0x7f042000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.655+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420005230 con 0x7f04240ccd60 2024-03-23T15:52:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.655+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0424151fd0 con 0x7f04240ccd60 2024-03-23T15:52:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.655+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04240ca3f0 con 0x7f04240ccd60 2024-03-23T15:52:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.655+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0420005390 con 0x7f04240ccd60 2024-03-23T15:52:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.655+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420007220 con 0x7f04240ccd60 2024-03-23T15:52:46.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.656+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0404005290 con 0x7f04240ccd60 2024-03-23T15:52:46.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.927+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0420007380 con 0x7f04240ccd60 2024-03-23T15:52:46.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.927+0000 7f0419ffb640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f040806b810 0x7f040806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:46.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.928+0000 7f0428c4a640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f040806b810 0x7f040806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:46.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.928+0000 7f0428c4a640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f040806b810 0x7f040806dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f041402b9c0 tx=0x7f0414008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:46.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.934+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0420058620 con 0x7f04240ccd60 2024-03-23T15:52:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:46.948+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f04200548c0 con 0x7f04240ccd60 2024-03-23T15:52:47.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.171+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04040058f0 con 0x7f04240ccd60 2024-03-23T15:52:47.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.190+0000 7f0419ffb640 1 -- 172.21.15.129:0/1781678246 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f04040058f0 con 0x7f04240ccd60 2024-03-23T15:52:47.384 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:47.384 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:47.385 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:47.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f040806b810 msgr2=0x7f040806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:47.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f040806b810 0x7f040806dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f041402b9c0 tx=0x7f0414008030 comp rx=0 tx=0).stop 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ccd60 msgr2=0x7f04241f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ccd60 0x7f04241f0a60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f042000c410 tx=0x7f042000c8d0 comp rx=0 tx=0).stop 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 shutdown_connections 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04240cef40 0x7f04240cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04240ceb00 0x7f0424159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f040806b810 0x7f040806dcd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 --2- 172.21.15.129:0/1781678246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04240ccd60 0x7f04241f0a60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 >> 172.21.15.129:0/1781678246 conn(0x7f04240b1840 msgr2=0x7f0424153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 shutdown_connections 2024-03-23T15:52:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:47.192+0000 7f042b6af640 1 -- 172.21.15.129:0/1781678246 wait complete. 2024-03-23T15:52:47.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:47.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:48.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.094+0000 7f375bfe2640 1 Processor -- start 2024-03-23T15:52:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.094+0000 7f375bfe2640 1 -- start start 2024-03-23T15:52:49.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.095+0000 7f375bfe2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f375415dc60 0x7f375406cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.095+0000 7f375bfe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415e040 0x7f37540cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.095+0000 7f375bfe2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f375406f570 0x7f37540cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.095+0000 7f375bfe2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f375406d2c0 con 0x7f375415dc60 2024-03-23T15:52:49.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.095+0000 7f375bfe2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f375415e740 con 0x7f375406f570 2024-03-23T15:52:49.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.095+0000 7f375bfe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f37540ce2e0 con 0x7f375415e040 2024-03-23T15:52:49.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.096+0000 7f375a57f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f375406f570 0x7f37540cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.096+0000 7f375957d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415e040 0x7f37540cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.096+0000 7f3759d7e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f375415dc60 0x7f375406cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.096+0000 7f375a57f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f375406f570 0x7f37540cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40064/0 (socket says 172.21.15.129:40064) 2024-03-23T15:52:49.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.096+0000 7f375a57f640 1 -- 172.21.15.129:0/1452381870 learned_addr learned my addr 172.21.15.129:0/1452381870 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:49.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.099+0000 7f375a57f640 1 -- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415e040 msgr2=0x7f37540cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:49.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.099+0000 7f375a57f640 1 --2- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415e040 0x7f37540cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.099+0000 7f375a57f640 1 -- 172.21.15.129:0/1452381870 >> v2:172.21.15.129:3300/0 conn(0x7f375415dc60 msgr2=0x7f375406cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:49.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.099+0000 7f375a57f640 1 --2- 172.21.15.129:0/1452381870 >> v2:172.21.15.129:3300/0 conn(0x7f375415dc60 0x7f375406cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.099+0000 7f375a57f640 1 -- 172.21.15.129:0/1452381870 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37540ccd00 con 0x7f375406f570 2024-03-23T15:52:49.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.102+0000 7f375a57f640 1 --2- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540cb7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f37540cc8c0 tx=0x7f374c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a41d038449db86 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:49.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.107+0000 7f3758d7c640 1 -- 172.21.15.129:0/1452381870 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c02c9f0 con 0x7f375406f570 2024-03-23T15:52:49.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.107+0000 7f3758d7c640 1 -- 172.21.15.129:0/1452381870 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374c02cb50 con 0x7f375406f570 2024-03-23T15:52:49.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.107+0000 7f3758d7c640 1 -- 172.21.15.129:0/1452381870 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c0343f0 con 0x7f375406f570 2024-03-23T15:52:49.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 -- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 msgr2=0x7f37540cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:49.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540cb7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f37540cc8c0 tx=0x7f374c02be50 comp rx=0 tx=0).stop 2024-03-23T15:52:49.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 -- 172.21.15.129:0/1452381870 shutdown_connections 2024-03-23T15:52:49.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540cb7c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1452381870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415e040 0x7f37540cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1452381870 >> v2:172.21.15.129:3300/0 conn(0x7f375415dc60 0x7f375406cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 -- 172.21.15.129:0/1452381870 >> 172.21.15.129:0/1452381870 conn(0x7f3754071300 msgr2=0x7f37540716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:49.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 -- 172.21.15.129:0/1452381870 shutdown_connections 2024-03-23T15:52:49.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.108+0000 7f375bfe2640 1 -- 172.21.15.129:0/1452381870 wait complete. 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.109+0000 7f375bfe2640 1 Processor -- start 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.109+0000 7f375bfe2640 1 -- start start 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.110+0000 7f375bfe2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.110+0000 7f375bfe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415dc60 0x7f3754157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.110+0000 7f375bfe2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375415e040 0x7f3754157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.110+0000 7f375bfe2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f37541f0960 con 0x7f375415e040 2024-03-23T15:52:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.110+0000 7f375bfe2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f375415a540 con 0x7f375406f570 2024-03-23T15:52:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.110+0000 7f375bfe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f37541efde0 con 0x7f375415dc60 2024-03-23T15:52:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.111+0000 7f3759d7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.111+0000 7f375a57f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375415e040 0x7f3754157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.111+0000 7f375957d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415dc60 0x7f3754157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.111+0000 7f3759d7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40072/0 (socket says 172.21.15.129:40072) 2024-03-23T15:52:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.111+0000 7f3759d7e640 1 -- 172.21.15.129:0/1756748220 learned_addr learned my addr 172.21.15.129:0/1756748220 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.116+0000 7f375a57f640 1 -- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415dc60 msgr2=0x7f3754157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.116+0000 7f375a57f640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415dc60 0x7f3754157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.116+0000 7f375a57f640 1 -- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 msgr2=0x7f37540c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.116+0000 7f375a57f640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.116+0000 7f375a57f640 1 -- 172.21.15.129:0/1756748220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c009b30 con 0x7f375415e040 2024-03-23T15:52:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.116+0000 7f3759d7e640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.123+0000 7f375a57f640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375415e040 0x7f3754157a60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f375415e880 tx=0x7f374c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.131+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c009510 con 0x7f375415e040 2024-03-23T15:52:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.131+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374c03e450 con 0x7f375415e040 2024-03-23T15:52:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.131+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c03e800 con 0x7f375415e040 2024-03-23T15:52:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.131+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37541f00c0 con 0x7f375415e040 2024-03-23T15:52:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.131+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37541f0680 con 0x7f375415e040 2024-03-23T15:52:49.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.132+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3754157fa0 con 0x7f375415e040 2024-03-23T15:52:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.419+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f374c03e960 con 0x7f375415e040 2024-03-23T15:52:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.419+0000 7f374a7fc640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f373806b840 0x7f373806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.420+0000 7f3759d7e640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f373806b840 0x7f373806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.420+0000 7f3759d7e640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f373806b840 0x7f373806dd00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ed0 tx=0x7f3750009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:49.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.441+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f374c07bb30 con 0x7f375415e040 2024-03-23T15:52:49.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.442+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f374c077dd0 con 0x7f375415e040 2024-03-23T15:52:49.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.689+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3754075740 con 0x7f375415e040 2024-03-23T15:52:49.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.703+0000 7f374a7fc640 1 -- 172.21.15.129:0/1756748220 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f374c043020 con 0x7f375415e040 2024-03-23T15:52:49.705 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:49.706 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:49.706 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f373806b840 msgr2=0x7f373806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f373806b840 0x7f373806dd00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ed0 tx=0x7f3750009190 comp rx=0 tx=0).stop 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375415e040 msgr2=0x7f3754157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375415e040 0x7f3754157a60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f375415e880 tx=0x7f374c004a40 comp rx=0 tx=0).stop 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 shutdown_connections 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375415e040 0x7f3754157a60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f375415dc60 0x7f3754157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f373806b840 0x7f373806dd00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 --2- 172.21.15.129:0/1756748220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f375406f570 0x7f37540c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:49.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 >> 172.21.15.129:0/1756748220 conn(0x7f3754071300 msgr2=0x7f375415b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:49.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 shutdown_connections 2024-03-23T15:52:49.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:49.706+0000 7f375bfe2640 1 -- 172.21.15.129:0/1756748220 wait complete. 2024-03-23T15:52:50.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:50.325 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:51.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.425+0000 7f1a9b284640 1 Processor -- start 2024-03-23T15:52:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.425+0000 7f1a9b284640 1 -- start start 2024-03-23T15:52:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.426+0000 7f1a9b284640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1a9406c410 0x7f1a9415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.426+0000 7f1a9b284640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9415de40 0x7f1a9415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.426+0000 7f1a9b284640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a9406c830 con 0x7f1a9406c030 2024-03-23T15:52:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.426+0000 7f1a9b284640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1a940cc370 con 0x7f1a9406c410 2024-03-23T15:52:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.426+0000 7f1a9b284640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9406cae0 con 0x7f1a9415de40 2024-03-23T15:52:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.427+0000 7f1a9881f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1a9406c410 0x7f1a9415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.427+0000 7f1a99821640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9415de40 0x7f1a9415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.427+0000 7f1a99020640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f1a9406c030 0x7f1a940cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56750/0 (socket says 172.21.15.129:56750) 2024-03-23T15:52:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.427+0000 7f1a99020640 1 -- 172.21.15.129:0/1362561821 learned_addr learned my addr 172.21.15.129:0/1362561821 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.429+0000 7f1a8bfff640 1 -- 172.21.15.129:0/1362561821 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932764982 0 0) 0x7f1a9406c830 con 0x7f1a9406c030 2024-03-23T15:52:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.429+0000 7f1a8bfff640 1 -- 172.21.15.129:0/1362561821 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a80003520 con 0x7f1a9406c030 2024-03-23T15:52:51.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.430+0000 7f1a9881f640 1 -- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9415de40 msgr2=0x7f1a9415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:51.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.430+0000 7f1a9881f640 1 --2- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9415de40 0x7f1a9415e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.430+0000 7f1a9881f640 1 -- 172.21.15.129:0/1362561821 >> v1:172.21.15.129:6789/0 conn(0x7f1a9406c030 legacy=0x7f1a940cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:51.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.430+0000 7f1a9881f640 1 -- 172.21.15.129:0/1362561821 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a940cabc0 con 0x7f1a9406c410 2024-03-23T15:52:51.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.430+0000 7f1a99821640 1 --2- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9415de40 0x7f1a9415e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:52:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.433+0000 7f1a9881f640 1 --2- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9406c410 0x7f1a9415d900 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1a840097a0 tx=0x7f1a8402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71878ca3ae280e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:51.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.437+0000 7f1a8bfff640 1 -- 172.21.15.129:0/1362561821 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8402d040 con 0x7f1a9406c410 2024-03-23T15:52:51.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.438+0000 7f1a8bfff640 1 -- 172.21.15.129:0/1362561821 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a84031810 con 0x7f1a9406c410 2024-03-23T15:52:51.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.438+0000 7f1a8bfff640 1 -- 172.21.15.129:0/1362561821 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84031b80 con 0x7f1a9406c410 2024-03-23T15:52:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 -- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9406c410 msgr2=0x7f1a9415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 --2- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9406c410 0x7f1a9415d900 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1a840097a0 tx=0x7f1a8402b8b0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 -- 172.21.15.129:0/1362561821 shutdown_connections 2024-03-23T15:52:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 --2- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9415de40 0x7f1a9415e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 --2- 172.21.15.129:0/1362561821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9406c410 0x7f1a9415d900 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 -- 172.21.15.129:0/1362561821 >> 172.21.15.129:0/1362561821 conn(0x7f1a940cfb20 msgr2=0x7f1a940cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 -- 172.21.15.129:0/1362561821 shutdown_connections 2024-03-23T15:52:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.439+0000 7f1a9b284640 1 -- 172.21.15.129:0/1362561821 wait complete. 2024-03-23T15:52:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.440+0000 7f1a9b284640 1 Processor -- start 2024-03-23T15:52:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.440+0000 7f1a9b284640 1 -- start start 2024-03-23T15:52:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9b284640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9406c030 0x7f1a940b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9b284640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9406c410 0x7f1a9414f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9b284640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 0x7f1a941f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9b284640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94157150 con 0x7f1a9406c410 2024-03-23T15:52:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9b284640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94155bf0 con 0x7f1a9415de40 2024-03-23T15:52:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9b284640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a941f0470 con 0x7f1a9406c030 2024-03-23T15:52:51.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a99821640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 0x7f1a941f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:51.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a99020640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9406c030 0x7f1a940b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:51.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.441+0000 7f1a9881f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9406c410 0x7f1a9414f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:51.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.442+0000 7f1a99821640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 0x7f1a941f3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52056/0 (socket says 172.21.15.129:52056) 2024-03-23T15:52:51.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.442+0000 7f1a99821640 1 -- 172.21.15.129:0/3174236253 learned_addr learned my addr 172.21.15.129:0/3174236253 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:51.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.446+0000 7f1a99821640 1 -- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9406c030 msgr2=0x7f1a940b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:51.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.446+0000 7f1a99821640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9406c030 0x7f1a940b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.446+0000 7f1a99821640 1 -- 172.21.15.129:0/3174236253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9406c410 msgr2=0x7f1a9414f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:51.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.446+0000 7f1a99821640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9406c410 0x7f1a9414f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.447+0000 7f1a99821640 1 -- 172.21.15.129:0/3174236253 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c009a70 con 0x7f1a9415de40 2024-03-23T15:52:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.450+0000 7f1a99821640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 0x7f1a941f3430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00efc0 tx=0x7f1a8c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.454+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c009cd0 con 0x7f1a9415de40 2024-03-23T15:52:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.454+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8c00f040 con 0x7f1a9415de40 2024-03-23T15:52:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.454+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c003990 con 0x7f1a9415de40 2024-03-23T15:52:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.454+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a84009400 con 0x7f1a9415de40 2024-03-23T15:52:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.454+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941f0a10 con 0x7f1a9415de40 2024-03-23T15:52:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.455+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a9414f120 con 0x7f1a9415de40 2024-03-23T15:52:51.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.479+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a8c014480 con 0x7f1a9415de40 2024-03-23T15:52:51.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.479+0000 7f1a89ffb640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a7806b890 0x7f1a7806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:51.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.479+0000 7f1a99020640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a7806b890 0x7f1a7806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:51.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.480+0000 7f1a99020640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a7806b890 0x7f1a7806dd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a90005880 tx=0x7f1a90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:51.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.738+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1a8c015e50 con 0x7f1a9415de40 2024-03-23T15:52:51.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.750+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a8c055cf0 con 0x7f1a9415de40 2024-03-23T15:52:51.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.971+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a9414f120 con 0x7f1a9415de40 2024-03-23T15:52:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.992+0000 7f1a89ffb640 1 -- 172.21.15.129:0/3174236253 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1a9414f120 con 0x7f1a9415de40 2024-03-23T15:52:51.996 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:51.996 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.994+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a7806b890 msgr2=0x7f1a7806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.994+0000 7f1a9b284640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a7806b890 0x7f1a7806dd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a90005880 tx=0x7f1a90009040 comp rx=0 tx=0).stop 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.994+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 msgr2=0x7f1a941f3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.994+0000 7f1a9b284640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 0x7f1a941f3430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00efc0 tx=0x7f1a8c00c490 comp rx=0 tx=0).stop 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 shutdown_connections 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a7806b890 0x7f1a7806dd50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9415de40 0x7f1a941f3430 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9406c410 0x7f1a9414f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 --2- 172.21.15.129:0/3174236253 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9406c030 0x7f1a940b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 >> 172.21.15.129:0/3174236253 conn(0x7f1a940cfb20 msgr2=0x7f1a940c7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 shutdown_connections 2024-03-23T15:52:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:51.995+0000 7f1a9b284640 1 -- 172.21.15.129:0/3174236253 wait complete. 2024-03-23T15:52:52.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:52.613 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:53.614 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:53.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.715+0000 7fadd2f93640 1 Processor -- start 2024-03-23T15:52:53.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.715+0000 7fadd2f93640 1 -- start start 2024-03-23T15:52:53.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.715+0000 7fadd2f93640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fadcc1544b0 0x7fadcc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:53.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.716+0000 7fadd2f93640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fadcc0caa00 0x7fadcc06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:53.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.716+0000 7fadd2f93640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadcc0cbcb0 con 0x7fadcc06c0b0 2024-03-23T15:52:53.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.716+0000 7fadd2f93640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fadcc0c5be0 con 0x7fadcc1544b0 2024-03-23T15:52:53.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.716+0000 7fadd2f93640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fadcc15b280 con 0x7fadcc0caa00 2024-03-23T15:52:53.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.716+0000 7fadd0d2f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fadcc1544b0 0x7fadcc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:53.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.716+0000 7fadcbfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fadcc0caa00 0x7fadcc06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:53.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.717+0000 7fadd0d2f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fadcc1544b0 0x7fadcc15a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45540/0 (socket says 172.21.15.129:45540) 2024-03-23T15:52:53.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.717+0000 7fadd0d2f640 1 -- 172.21.15.129:0/2384957956 learned_addr learned my addr 172.21.15.129:0/2384957956 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.719+0000 7fadd0d2f640 1 -- 172.21.15.129:0/2384957956 >> v2:172.21.15.157:3300/0 conn(0x7fadcc0caa00 msgr2=0x7fadcc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.719+0000 7fadd0d2f640 1 --2- 172.21.15.129:0/2384957956 >> v2:172.21.15.157:3300/0 conn(0x7fadcc0caa00 0x7fadcc06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.719+0000 7fadd0d2f640 1 -- 172.21.15.129:0/2384957956 >> v1:172.21.15.157:6789/0 conn(0x7fadcc06c0b0 legacy=0x7fadcc0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.719+0000 7fadd0d2f640 1 -- 172.21.15.129:0/2384957956 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadcc15a9e0 con 0x7fadcc1544b0 2024-03-23T15:52:53.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.721+0000 7fadd0d2f640 1 --2- 172.21.15.129:0/2384957956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc1544b0 0x7fadcc15a2e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fadcc06ce60 tx=0x7fadc002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448086fd923c0aa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:53.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.723+0000 7fadcb7fe640 1 -- 172.21.15.129:0/2384957956 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc002f850 con 0x7fadcc1544b0 2024-03-23T15:52:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.725+0000 7fadcb7fe640 1 -- 172.21.15.129:0/2384957956 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc002f9b0 con 0x7fadcc1544b0 2024-03-23T15:52:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.725+0000 7fadcb7fe640 1 -- 172.21.15.129:0/2384957956 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc002fd40 con 0x7fadcc1544b0 2024-03-23T15:52:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 -- 172.21.15.129:0/2384957956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc1544b0 msgr2=0x7fadcc15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 --2- 172.21.15.129:0/2384957956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc1544b0 0x7fadcc15a2e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fadcc06ce60 tx=0x7fadc002ba30 comp rx=0 tx=0).stop 2024-03-23T15:52:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 -- 172.21.15.129:0/2384957956 shutdown_connections 2024-03-23T15:52:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 --2- 172.21.15.129:0/2384957956 >> v2:172.21.15.157:3300/0 conn(0x7fadcc0caa00 0x7fadcc06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:53.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 --2- 172.21.15.129:0/2384957956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc1544b0 0x7fadcc15a2e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:53.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 -- 172.21.15.129:0/2384957956 >> 172.21.15.129:0/2384957956 conn(0x7fadcc061b50 msgr2=0x7fadcc061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:53.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 -- 172.21.15.129:0/2384957956 shutdown_connections 2024-03-23T15:52:53.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.726+0000 7fadd2f93640 1 -- 172.21.15.129:0/2384957956 wait complete. 2024-03-23T15:52:53.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.727+0000 7fadd2f93640 1 Processor -- start 2024-03-23T15:52:53.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.727+0000 7fadd2f93640 1 -- start start 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd2f93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 0x7fadcc0cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd2f93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadcc0caa00 0x7fadcc1f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd2f93640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadcc1544b0 0x7fadcc1f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd2f93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc1eff60 con 0x7fadcc06c0b0 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd2f93640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc154e80 con 0x7fadcc1544b0 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd2f93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fadcc1553b0 con 0x7fadcc0caa00 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.728+0000 7fadd0d2f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 0x7fadcc0cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.729+0000 7fadd1530640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadcc1544b0 0x7fadcc1f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.729+0000 7fadcbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadcc0caa00 0x7fadcc1f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.729+0000 7fadd0d2f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 0x7fadcc0cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45552/0 (socket says 172.21.15.129:45552) 2024-03-23T15:52:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.729+0000 7fadd0d2f640 1 -- 172.21.15.129:0/3121000982 learned_addr learned my addr 172.21.15.129:0/3121000982 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:53.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.732+0000 7fadd0d2f640 1 -- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadcc0caa00 msgr2=0x7fadcc1f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:53.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.732+0000 7fadd0d2f640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadcc0caa00 0x7fadcc1f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:53.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.732+0000 7fadd0d2f640 1 -- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadcc1544b0 msgr2=0x7fadcc1f08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:53.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.732+0000 7fadd0d2f640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadcc1544b0 0x7fadcc1f08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:53.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.732+0000 7fadd0d2f640 1 -- 172.21.15.129:0/3121000982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc00094d0 con 0x7fadcc06c0b0 2024-03-23T15:52:53.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.735+0000 7fadd0d2f640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 0x7fadcc0cdcb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadc0000c00 tx=0x7fadc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:53.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.737+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0037bc0 con 0x7fadcc06c0b0 2024-03-23T15:52:53.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.738+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadcc1550d0 con 0x7fadcc06c0b0 2024-03-23T15:52:53.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.738+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadcc156ec0 con 0x7fadcc06c0b0 2024-03-23T15:52:53.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:53.738+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadcc157080 con 0x7fadcc06c0b0 2024-03-23T15:52:54.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.007+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc0037d20 con 0x7fadcc06c0b0 2024-03-23T15:52:54.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.007+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0042680 con 0x7fadcc06c0b0 2024-03-23T15:52:54.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.008+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fadc0042820 con 0x7fadcc06c0b0 2024-03-23T15:52:54.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.008+0000 7fadc97fa640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fadb006b810 0x7fadb006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:54.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.009+0000 7fadcbfff640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fadb006b810 0x7fadb006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:54.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.009+0000 7fadcbfff640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fadb006b810 0x7fadb006dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fadcc1edb90 tx=0x7fadb8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:54.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.021+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fadc003c070 con 0x7fadcc06c0b0 2024-03-23T15:52:54.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.022+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fadc0077c60 con 0x7fadcc06c0b0 2024-03-23T15:52:54.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.240+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadcc1f0200 con 0x7fadcc06c0b0 2024-03-23T15:52:54.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.253+0000 7fadc97fa640 1 -- 172.21.15.129:0/3121000982 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fadc0051020 con 0x7fadcc06c0b0 2024-03-23T15:52:54.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:54.256 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:54.256 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:54.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fadb006b810 msgr2=0x7fadb006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:54.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fadb006b810 0x7fadb006dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fadcc1edb90 tx=0x7fadb8009190 comp rx=0 tx=0).stop 2024-03-23T15:52:54.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 msgr2=0x7fadcc0cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:54.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 0x7fadcc0cdcb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadc0000c00 tx=0x7fadc0004730 comp rx=0 tx=0).stop 2024-03-23T15:52:54.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 shutdown_connections 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fadb006b810 0x7fadb006dcd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadcc1544b0 0x7fadcc1f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadcc0caa00 0x7fadcc1f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 --2- 172.21.15.129:0/3121000982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadcc06c0b0 0x7fadcc0cdcb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 >> 172.21.15.129:0/3121000982 conn(0x7fadcc061b50 msgr2=0x7fadcc0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 shutdown_connections 2024-03-23T15:52:54.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:54.255+0000 7fadd2f93640 1 -- 172.21.15.129:0/3121000982 wait complete. 2024-03-23T15:52:54.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:54.874 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:55.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:55.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.973+0000 7f44b4c6f640 1 Processor -- start 2024-03-23T15:52:55.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.973+0000 7f44b4c6f640 1 -- start start 2024-03-23T15:52:55.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.974+0000 7f44b4c6f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f44b0154430 0x7f44b0071b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:55.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.974+0000 7f44b4c6f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b015d810 con 0x7f44b00728c0 2024-03-23T15:52:55.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.974+0000 7f44b4c6f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b006f0f0 con 0x7f44b015a220 2024-03-23T15:52:55.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.974+0000 7f44b4c6f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f44b015a780 con 0x7f44b0154430 2024-03-23T15:52:55.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.975+0000 7f44aed9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f44b00728c0 0x7f44b0164ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56766/0 (socket says 172.21.15.129:56766) 2024-03-23T15:52:55.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.975+0000 7f44aed9d640 1 -- 172.21.15.129:0/2146763779 learned_addr learned my addr 172.21.15.129:0/2146763779 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:55.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.975+0000 7f44ae59c640 1 --2- 172.21.15.129:0/2146763779 >> v2:172.21.15.157:3300/0 conn(0x7f44b0154430 0x7f44b0071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:55.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.977+0000 7f44ad59a640 1 -- 172.21.15.129:0/2146763779 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176177194 0 0) 0x7f44b015d810 con 0x7f44b00728c0 2024-03-23T15:52:55.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.977+0000 7f44ad59a640 1 -- 172.21.15.129:0/2146763779 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4498003520 con 0x7f44b00728c0 2024-03-23T15:52:55.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.978+0000 7f44ae59c640 1 -- 172.21.15.129:0/2146763779 >> v1:172.21.15.157:6789/0 conn(0x7f44b015a220 legacy=0x7f44b015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:55.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.978+0000 7f44ae59c640 1 -- 172.21.15.129:0/2146763779 >> v1:172.21.15.129:6789/0 conn(0x7f44b00728c0 legacy=0x7f44b0164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:55.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.978+0000 7f44ae59c640 1 -- 172.21.15.129:0/2146763779 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b015aa60 con 0x7f44b0154430 2024-03-23T15:52:55.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.981+0000 7f44ae59c640 1 --2- 172.21.15.129:0/2146763779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b0154430 0x7f44b0071b60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f44a00097a0 tx=0x7f44a002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67cf8d8f3770cb10 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:55.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.987+0000 7f44ad59a640 1 -- 172.21.15.129:0/2146763779 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0031830 con 0x7f44b0154430 2024-03-23T15:52:55.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.987+0000 7f44ad59a640 1 -- 172.21.15.129:0/2146763779 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a0031990 con 0x7f44b0154430 2024-03-23T15:52:55.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.987+0000 7f44ad59a640 1 -- 172.21.15.129:0/2146763779 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0031d20 con 0x7f44b0154430 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 -- 172.21.15.129:0/2146763779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b0154430 msgr2=0x7f44b0071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/2146763779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b0154430 0x7f44b0071b60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f44a00097a0 tx=0x7f44a002b8d0 comp rx=0 tx=0).stop 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 -- 172.21.15.129:0/2146763779 shutdown_connections 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/2146763779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b0154430 0x7f44b0071b60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 -- 172.21.15.129:0/2146763779 >> 172.21.15.129:0/2146763779 conn(0x7f44b00cbb20 msgr2=0x7f44b00cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 -- 172.21.15.129:0/2146763779 shutdown_connections 2024-03-23T15:52:55.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.988+0000 7f44b4c6f640 1 -- 172.21.15.129:0/2146763779 wait complete. 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 Processor -- start 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 -- start start 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b00728c0 0x7f44b00c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b0154430 0x7f44b01504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44b015a220 0x7f44b0146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44b00cabe0 con 0x7f44b015a220 2024-03-23T15:52:55.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f44b014c9a0 con 0x7f44b00728c0 2024-03-23T15:52:55.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.990+0000 7f44b4c6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f44b0146660 con 0x7f44b0154430 2024-03-23T15:52:55.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.991+0000 7f44aed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44b015a220 0x7f44b0146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:55.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.991+0000 7f44ae59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b00728c0 0x7f44b00c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:55.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.991+0000 7f44add9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b0154430 0x7f44b01504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:55.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.991+0000 7f44aed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44b015a220 0x7f44b0146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45564/0 (socket says 172.21.15.129:45564) 2024-03-23T15:52:55.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.991+0000 7f44aed9d640 1 -- 172.21.15.129:0/755632688 learned_addr learned my addr 172.21.15.129:0/755632688 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:55.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.995+0000 7f44add9b640 1 -- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b00728c0 msgr2=0x7f44b00c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:55.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.995+0000 7f44add9b640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b00728c0 0x7f44b00c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:55.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.995+0000 7f44add9b640 1 -- 172.21.15.129:0/755632688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44b015a220 msgr2=0x7f44b0146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:55.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.995+0000 7f44add9b640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44b015a220 0x7f44b0146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:55.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.995+0000 7f44add9b640 1 -- 172.21.15.129:0/755632688 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a0009400 con 0x7f44b0154430 2024-03-23T15:52:56.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:55.998+0000 7f44add9b640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b0154430 0x7f44b01504a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44a400b3e0 tx=0x7f44a400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:56.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.003+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4011830 con 0x7f44b0154430 2024-03-23T15:52:56.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.003+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a4011990 con 0x7f44b0154430 2024-03-23T15:52:56.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.003+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b01f3b50 con 0x7f44b0154430 2024-03-23T15:52:56.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.003+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4011d40 con 0x7f44b0154430 2024-03-23T15:52:56.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.003+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b0151310 con 0x7f44b0154430 2024-03-23T15:52:56.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.004+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b015e670 con 0x7f44b0154430 2024-03-23T15:52:56.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.028+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f44a401a400 con 0x7f44b0154430 2024-03-23T15:52:56.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.028+0000 7f449effd640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f449006b890 0x7f449006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:56.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.029+0000 7f44ae59c640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f449006b890 0x7f449006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:56.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.029+0000 7f44ae59c640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f449006b890 0x7f449006dd50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f44a0036000 tx=0x7f44a003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.294+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f44a4058dc0 con 0x7f44b0154430 2024-03-23T15:52:56.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.305+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44a4055060 con 0x7f44b0154430 2024-03-23T15:52:56.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.538+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44b015e670 con 0x7f44b0154430 2024-03-23T15:52:56.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.554+0000 7f449effd640 1 -- 172.21.15.129:0/755632688 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f44b015e670 con 0x7f44b0154430 2024-03-23T15:52:56.557 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:56.557 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:56.558 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:56.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f449006b890 msgr2=0x7f449006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:56.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f449006b890 0x7f449006dd50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f44a0036000 tx=0x7f44a003c040 comp rx=0 tx=0).stop 2024-03-23T15:52:56.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b0154430 msgr2=0x7f44b01504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b0154430 0x7f44b01504a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44a400b3e0 tx=0x7f44a400b8a0 comp rx=0 tx=0).stop 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 shutdown_connections 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44b015a220 0x7f44b0146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b0154430 0x7f44b01504a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f449006b890 0x7f449006dd50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 --2- 172.21.15.129:0/755632688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44b00728c0 0x7f44b00c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 >> 172.21.15.129:0/755632688 conn(0x7f44b00cbb20 msgr2=0x7f44b00cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 shutdown_connections 2024-03-23T15:52:56.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:56.556+0000 7f44b4c6f640 1 -- 172.21.15.129:0/755632688 wait complete. 2024-03-23T15:52:57.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:57.180 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:52:58.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:52:58.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.281+0000 7fa2feb8a640 1 Processor -- start 2024-03-23T15:52:58.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.282+0000 7fa2feb8a640 1 -- start start 2024-03-23T15:52:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.282+0000 7fa2feb8a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa2f80ca580 0x7fa2f806c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.282+0000 7fa2feb8a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f815d880 con 0x7fa2f8154270 2024-03-23T15:52:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.282+0000 7fa2feb8a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f80ca9d0 con 0x7fa2f806c650 2024-03-23T15:52:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.282+0000 7fa2feb8a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f80c4ae0 con 0x7fa2f80ca580 2024-03-23T15:52:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.283+0000 7fa2fc926640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa2f8154270 0x7fa2f815ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56772/0 (socket says 172.21.15.129:56772) 2024-03-23T15:52:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.283+0000 7fa2fc926640 1 -- 172.21.15.129:0/458644238 learned_addr learned my addr 172.21.15.129:0/458644238 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:52:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.283+0000 7fa2f7fff640 1 --2- 172.21.15.129:0/458644238 >> v2:172.21.15.157:3300/0 conn(0x7fa2f80ca580 0x7fa2f806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.285+0000 7fa2f77fe640 1 -- 172.21.15.129:0/458644238 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795628205 0 0) 0x7fa2f815d880 con 0x7fa2f8154270 2024-03-23T15:52:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.285+0000 7fa2f77fe640 1 -- 172.21.15.129:0/458644238 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2e8003520 con 0x7fa2f8154270 2024-03-23T15:52:58.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.287+0000 7fa2f7fff640 1 -- 172.21.15.129:0/458644238 >> v1:172.21.15.157:6789/0 conn(0x7fa2f806c650 legacy=0x7fa2f80c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:52:58.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.287+0000 7fa2f7fff640 1 -- 172.21.15.129:0/458644238 >> v1:172.21.15.129:6789/0 conn(0x7fa2f8154270 legacy=0x7fa2f815ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:58.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.287+0000 7fa2f7fff640 1 -- 172.21.15.129:0/458644238 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f80cd5e0 con 0x7fa2f80ca580 2024-03-23T15:52:58.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.289+0000 7fa2f7fff640 1 --2- 172.21.15.129:0/458644238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f80ca580 0x7fa2f806c110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40097a0 tx=0x7fa2e402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2572f36f25ccd4cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.295+0000 7fa2f77fe640 1 -- 172.21.15.129:0/458644238 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e402f050 con 0x7fa2f80ca580 2024-03-23T15:52:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.295+0000 7fa2f77fe640 1 -- 172.21.15.129:0/458644238 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e4031700 con 0x7fa2f80ca580 2024-03-23T15:52:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.295+0000 7fa2f77fe640 1 -- 172.21.15.129:0/458644238 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4031a90 con 0x7fa2f80ca580 2024-03-23T15:52:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 -- 172.21.15.129:0/458644238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f80ca580 msgr2=0x7fa2f806c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/458644238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f80ca580 0x7fa2f806c110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40097a0 tx=0x7fa2e402b7a0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 -- 172.21.15.129:0/458644238 shutdown_connections 2024-03-23T15:52:58.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/458644238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f80ca580 0x7fa2f806c110 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 -- 172.21.15.129:0/458644238 >> 172.21.15.129:0/458644238 conn(0x7fa2f806d290 msgr2=0x7fa2f8072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:58.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 -- 172.21.15.129:0/458644238 shutdown_connections 2024-03-23T15:52:58.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.296+0000 7fa2feb8a640 1 -- 172.21.15.129:0/458644238 wait complete. 2024-03-23T15:52:58.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.297+0000 7fa2feb8a640 1 Processor -- start 2024-03-23T15:52:58.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.297+0000 7fa2feb8a640 1 -- start start 2024-03-23T15:52:58.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.298+0000 7fa2feb8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f806c650 0x7fa2f815c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:58.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.298+0000 7fa2feb8a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2f80ca580 0x7fa2f8153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:58.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.298+0000 7fa2feb8a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 0x7fa2f80b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:58.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.298+0000 7fa2feb8a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f815d000 con 0x7fa2f8154270 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.298+0000 7fa2feb8a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f8152e00 con 0x7fa2f806c650 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.298+0000 7fa2feb8a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f8149340 con 0x7fa2f80ca580 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.299+0000 7fa2fd127640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 0x7fa2f80b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.299+0000 7fa2f7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2f80ca580 0x7fa2f8153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.299+0000 7fa2fc926640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f806c650 0x7fa2f815c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.299+0000 7fa2fd127640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 0x7fa2f80b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45570/0 (socket says 172.21.15.129:45570) 2024-03-23T15:52:58.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.299+0000 7fa2fd127640 1 -- 172.21.15.129:0/3684383711 learned_addr learned my addr 172.21.15.129:0/3684383711 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:52:58.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.303+0000 7fa2fd127640 1 -- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2f80ca580 msgr2=0x7fa2f8153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:58.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.303+0000 7fa2fd127640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2f80ca580 0x7fa2f8153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.303+0000 7fa2fd127640 1 -- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f806c650 msgr2=0x7fa2f815c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:58.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.303+0000 7fa2fd127640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f806c650 0x7fa2f815c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.303+0000 7fa2fd127640 1 -- 172.21.15.129:0/3684383711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e4009400 con 0x7fa2f8154270 2024-03-23T15:52:58.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.305+0000 7fa2fd127640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 0x7fa2f80b3210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000b9c0 tx=0x7fa2f000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.309+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000cb40 con 0x7fa2f8154270 2024-03-23T15:52:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.309+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f000cca0 con 0x7fa2f8154270 2024-03-23T15:52:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.309+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0014680 con 0x7fa2f8154270 2024-03-23T15:52:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.309+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f815a000 con 0x7fa2f8154270 2024-03-23T15:52:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.310+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f815d530 con 0x7fa2f8154270 2024-03-23T15:52:58.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.311+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f8075740 con 0x7fa2f8154270 2024-03-23T15:52:58.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.566+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa2f001d3b0 con 0x7fa2f8154270 2024-03-23T15:52:58.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.566+0000 7fa2f57fa640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa2dc06b860 0x7fa2dc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:52:58.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.567+0000 7fa2fc926640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa2dc06b860 0x7fa2dc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:52:58.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.567+0000 7fa2fc926640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa2dc06b860 0x7fa2dc06dd20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec005880 tx=0x7fa2ec009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:52:58.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.583+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa2f0058dc0 con 0x7fa2f8154270 2024-03-23T15:52:58.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.583+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa2f0055060 con 0x7fa2f8154270 2024-03-23T15:52:58.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.840+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2f80c9f30 con 0x7fa2f8154270 2024-03-23T15:52:58.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.855+0000 7fa2f57fa640 1 -- 172.21.15.129:0/3684383711 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa2f0019020 con 0x7fa2f8154270 2024-03-23T15:52:58.858 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:52:58.858 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:52:58.859 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa2dc06b860 msgr2=0x7fa2dc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa2dc06b860 0x7fa2dc06dd20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec005880 tx=0x7fa2ec009040 comp rx=0 tx=0).stop 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 msgr2=0x7fa2f80b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 0x7fa2f80b3210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000b9c0 tx=0x7fa2f000be80 comp rx=0 tx=0).stop 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 shutdown_connections 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2f8154270 0x7fa2f80b3210 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2f80ca580 0x7fa2f8153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa2dc06b860 0x7fa2dc06dd20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 --2- 172.21.15.129:0/3684383711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2f806c650 0x7fa2f815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:52:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 >> 172.21.15.129:0/3684383711 conn(0x7fa2f806d290 msgr2=0x7fa2f80cbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:52:58.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 shutdown_connections 2024-03-23T15:52:58.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:52:58.857+0000 7fa2feb8a640 1 -- 172.21.15.129:0/3684383711 wait complete. 2024-03-23T15:52:59.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:52:59.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:00.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:00.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.576+0000 7fd662874640 1 Processor -- start 2024-03-23T15:53:00.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.576+0000 7fd662874640 1 -- start start 2024-03-23T15:53:00.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.576+0000 7fd662874640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c15dc60 0x7fd65c06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:00.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.576+0000 7fd662874640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd65c15e040 0x7fd65c0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:00.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.577+0000 7fd662874640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65c06d2c0 con 0x7fd65c06f570 2024-03-23T15:53:00.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.577+0000 7fd662874640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd65c15e740 con 0x7fd65c15e040 2024-03-23T15:53:00.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.577+0000 7fd662874640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c0ce2e0 con 0x7fd65c15dc60 2024-03-23T15:53:00.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.577+0000 7fd65b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd65c15e040 0x7fd65c0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:00.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.577+0000 7fd65bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c15dc60 0x7fd65c06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:00.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.578+0000 7fd65b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd65c15e040 0x7fd65c0cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41748/0 (socket says 172.21.15.129:41748) 2024-03-23T15:53:00.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.578+0000 7fd65b7fe640 1 -- 172.21.15.129:0/3721979720 learned_addr learned my addr 172.21.15.129:0/3721979720 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:00.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.580+0000 7fd65b7fe640 1 -- 172.21.15.129:0/3721979720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c15dc60 msgr2=0x7fd65c06cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:00.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.580+0000 7fd65b7fe640 1 --2- 172.21.15.129:0/3721979720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c15dc60 0x7fd65c06cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:00.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.580+0000 7fd65b7fe640 1 -- 172.21.15.129:0/3721979720 >> v1:172.21.15.129:6789/0 conn(0x7fd65c06f570 legacy=0x7fd65c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:00.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.580+0000 7fd65b7fe640 1 -- 172.21.15.129:0/3721979720 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c0ccd00 con 0x7fd65c15e040 2024-03-23T15:53:00.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.584+0000 7fd65b7fe640 1 --2- 172.21.15.129:0/3721979720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15e040 0x7fd65c0cdda0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd650009870 tx=0x7fd65002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6a140008e8977b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:00.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.591+0000 7fd65affd640 1 -- 172.21.15.129:0/3721979720 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650031640 con 0x7fd65c15e040 2024-03-23T15:53:00.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.591+0000 7fd65affd640 1 -- 172.21.15.129:0/3721979720 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6500317a0 con 0x7fd65c15e040 2024-03-23T15:53:00.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.591+0000 7fd65affd640 1 -- 172.21.15.129:0/3721979720 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650031b30 con 0x7fd65c15e040 2024-03-23T15:53:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 -- 172.21.15.129:0/3721979720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15e040 msgr2=0x7fd65c0cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 --2- 172.21.15.129:0/3721979720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15e040 0x7fd65c0cdda0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd650009870 tx=0x7fd65002b8f0 comp rx=0 tx=0).stop 2024-03-23T15:53:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 -- 172.21.15.129:0/3721979720 shutdown_connections 2024-03-23T15:53:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 --2- 172.21.15.129:0/3721979720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15e040 0x7fd65c0cdda0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 --2- 172.21.15.129:0/3721979720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c15dc60 0x7fd65c06cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 -- 172.21.15.129:0/3721979720 >> 172.21.15.129:0/3721979720 conn(0x7fd65c071300 msgr2=0x7fd65c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 -- 172.21.15.129:0/3721979720 shutdown_connections 2024-03-23T15:53:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.592+0000 7fd662874640 1 -- 172.21.15.129:0/3721979720 wait complete. 2024-03-23T15:53:00.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.594+0000 7fd662874640 1 Processor -- start 2024-03-23T15:53:00.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.594+0000 7fd662874640 1 -- start start 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.595+0000 7fd662874640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c06f570 0x7fd65c1f0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.595+0000 7fd662874640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 0x7fd65c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.595+0000 7fd662874640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c15e040 0x7fd65c1502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.595+0000 7fd662874640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c0c6e00 con 0x7fd65c15dc60 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.595+0000 7fd662874640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1548a0 con 0x7fd65c15e040 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.595+0000 7fd662874640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c154b80 con 0x7fd65c06f570 2024-03-23T15:53:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.596+0000 7fd65b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 0x7fd65c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.596+0000 7fd65b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 0x7fd65c0c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41762/0 (socket says 172.21.15.129:41762) 2024-03-23T15:53:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.596+0000 7fd65b7fe640 1 -- 172.21.15.129:0/1860539506 learned_addr learned my addr 172.21.15.129:0/1860539506 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.596+0000 7fd65bfff640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c06f570 0x7fd65c1f0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.596+0000 7fd660e11640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c15e040 0x7fd65c1502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.600+0000 7fd65b7fe640 1 -- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c06f570 msgr2=0x7fd65c1f0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.600+0000 7fd65b7fe640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c06f570 0x7fd65c1f0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.600+0000 7fd65b7fe640 1 -- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c15e040 msgr2=0x7fd65c1502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.600+0000 7fd65b7fe640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c15e040 0x7fd65c1502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.600+0000 7fd65b7fe640 1 -- 172.21.15.129:0/1860539506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6500094d0 con 0x7fd65c15dc60 2024-03-23T15:53:00.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.602+0000 7fd65b7fe640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 0x7fd65c0c5710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd650000c00 tx=0x7fd6500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:00.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.607+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65003a930 con 0x7fd65c15dc60 2024-03-23T15:53:00.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.607+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65003aa90 con 0x7fd65c15dc60 2024-03-23T15:53:00.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.607+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65003ae40 con 0x7fd65c15dc60 2024-03-23T15:53:00.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.607+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c154e60 con 0x7fd65c15dc60 2024-03-23T15:53:00.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.607+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c0c5c50 con 0x7fd65c15dc60 2024-03-23T15:53:00.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.608+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65c075740 con 0x7fd65c15dc60 2024-03-23T15:53:00.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.634+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd65002f050 con 0x7fd65c15dc60 2024-03-23T15:53:00.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.634+0000 7fd658ff9640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd64806b860 0x7fd64806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:00.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.634+0000 7fd65bfff640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd64806b860 0x7fd64806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.635+0000 7fd65bfff640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd64806b860 0x7fd64806dd20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009aa0 tx=0x7fd64c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:00.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.894+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd65007aba0 con 0x7fd65c15dc60 2024-03-23T15:53:00.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:00.895+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd650076e40 con 0x7fd65c15dc60 2024-03-23T15:53:01.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.168+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd65c0c8eb0 con 0x7fd65c15dc60 2024-03-23T15:53:01.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.189+0000 7fd658ff9640 1 -- 172.21.15.129:0/1860539506 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd650035090 con 0x7fd65c15dc60 2024-03-23T15:53:01.193 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:01.193 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:01.193 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd64806b860 msgr2=0x7fd64806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd64806b860 0x7fd64806dd20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009aa0 tx=0x7fd64c009190 comp rx=0 tx=0).stop 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 msgr2=0x7fd65c0c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 0x7fd65c0c5710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd650000c00 tx=0x7fd6500046c0 comp rx=0 tx=0).stop 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 shutdown_connections 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd64806b860 0x7fd64806dd20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:01.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c15e040 0x7fd65c1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:01.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c15dc60 0x7fd65c0c5710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:01.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 --2- 172.21.15.129:0/1860539506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c06f570 0x7fd65c1f0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:01.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 >> 172.21.15.129:0/1860539506 conn(0x7fd65c071300 msgr2=0x7fd65c06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:01.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 shutdown_connections 2024-03-23T15:53:01.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:01.192+0000 7fd662874640 1 -- 172.21.15.129:0/1860539506 wait complete. 2024-03-23T15:53:01.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:01.811 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:02.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:02.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.911+0000 7f08680fb640 1 Processor -- start 2024-03-23T15:53:02.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.912+0000 7f08680fb640 1 -- start start 2024-03-23T15:53:02.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.912+0000 7f08680fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ccd60 0x7f086015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:02.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.912+0000 7f08680fb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08600ceb00 0x7f086006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:02.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.912+0000 7f08680fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086015b290 con 0x7f08600cef40 2024-03-23T15:53:02.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.912+0000 7f08680fb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f08600ce800 con 0x7f08600ceb00 2024-03-23T15:53:02.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.912+0000 7f08680fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0860154a70 con 0x7f08600ccd60 2024-03-23T15:53:02.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.913+0000 7f0865696640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08600ceb00 0x7f086006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:02.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.913+0000 7f0866698640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f08600cef40 0x7f086015b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36062/0 (socket says 172.21.15.129:36062) 2024-03-23T15:53:02.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.913+0000 7f0866698640 1 -- 172.21.15.129:0/4208373992 learned_addr learned my addr 172.21.15.129:0/4208373992 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:02.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.913+0000 7f0865e97640 1 --2- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ccd60 0x7f086015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:02.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.916+0000 7f0864e95640 1 -- 172.21.15.129:0/4208373992 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594925413 0 0) 0x7f0858001490 con 0x7f08600cef40 2024-03-23T15:53:02.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.916+0000 7f0864e95640 1 -- 172.21.15.129:0/4208373992 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f084c003520 con 0x7f08600cef40 2024-03-23T15:53:02.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.917+0000 7f0865696640 1 -- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ccd60 msgr2=0x7f086015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:02.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.917+0000 7f0865696640 1 --2- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ccd60 0x7f086015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:02.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.917+0000 7f0865696640 1 -- 172.21.15.129:0/4208373992 >> v1:172.21.15.129:6789/0 conn(0x7f08600cef40 legacy=0x7f086015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:02.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.917+0000 7f0865696640 1 -- 172.21.15.129:0/4208373992 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08600cd950 con 0x7f08600ceb00 2024-03-23T15:53:02.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.920+0000 7f0865696640 1 --2- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600ceb00 0x7f086006c270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f08500097a0 tx=0x7f085002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3ac069af71dff4b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:02.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f0864e95640 1 -- 172.21.15.129:0/4208373992 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850030610 con 0x7f08600ceb00 2024-03-23T15:53:02.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f0864e95640 1 -- 172.21.15.129:0/4208373992 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0850030770 con 0x7f08600ceb00 2024-03-23T15:53:02.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f0864e95640 1 -- 172.21.15.129:0/4208373992 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850030b00 con 0x7f08600ceb00 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f08680fb640 1 -- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600ceb00 msgr2=0x7f086006c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f08680fb640 1 --2- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600ceb00 0x7f086006c270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f08500097a0 tx=0x7f085002b960 comp rx=0 tx=0).stop 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f08680fb640 1 -- 172.21.15.129:0/4208373992 shutdown_connections 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f08680fb640 1 --2- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600ceb00 0x7f086006c270 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f08680fb640 1 --2- 172.21.15.129:0/4208373992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ccd60 0x7f086015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.925+0000 7f08680fb640 1 -- 172.21.15.129:0/4208373992 >> 172.21.15.129:0/4208373992 conn(0x7f08600b1840 msgr2=0x7f08600b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.926+0000 7f08680fb640 1 -- 172.21.15.129:0/4208373992 shutdown_connections 2024-03-23T15:53:02.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.926+0000 7f08680fb640 1 -- 172.21.15.129:0/4208373992 wait complete. 2024-03-23T15:53:02.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 Processor -- start 2024-03-23T15:53:02.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 -- start start 2024-03-23T15:53:02.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08600ccd60 0x7f08601f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:02.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 0x7f08601e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:02.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600cef40 0x7f0860151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:02.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f08601e4dc0 con 0x7f08600ccd60 2024-03-23T15:53:02.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08601e5250 con 0x7f08600cef40 2024-03-23T15:53:02.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.927+0000 7f08680fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08601e3ed0 con 0x7f08600ceb00 2024-03-23T15:53:02.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.928+0000 7f0865696640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 0x7f08601e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:02.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.928+0000 7f0866698640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600cef40 0x7f0860151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:02.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.928+0000 7f0865e97640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08600ccd60 0x7f08601f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:02.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.928+0000 7f0866698640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600cef40 0x7f0860151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:57060/0 (socket says 172.21.15.129:57060) 2024-03-23T15:53:02.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.928+0000 7f0866698640 1 -- 172.21.15.129:0/1880212610 learned_addr learned my addr 172.21.15.129:0/1880212610 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:02.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.928+0000 7f0865696640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 0x7f08601e3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:37076/0 (socket says 172.21.15.129:37076) 2024-03-23T15:53:02.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.932+0000 7f0865696640 1 -- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600cef40 msgr2=0x7f0860151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:02.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.932+0000 7f0865696640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600cef40 0x7f0860151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:02.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.932+0000 7f0865696640 1 -- 172.21.15.129:0/1880212610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08600ccd60 msgr2=0x7f08601f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:02.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.932+0000 7f0865696640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08600ccd60 0x7f08601f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:02.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.932+0000 7f0865696640 1 -- 172.21.15.129:0/1880212610 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0850009400 con 0x7f08600ceb00 2024-03-23T15:53:02.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.935+0000 7f0865696640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 0x7f08601e3640 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0850033000 tx=0x7f0850004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:02.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.939+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085003e040 con 0x7f08600ceb00 2024-03-23T15:53:02.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.939+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0850039a40 con 0x7f08600ceb00 2024-03-23T15:53:02.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.939+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850039df0 con 0x7f08600ceb00 2024-03-23T15:53:02.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.939+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08601e41b0 con 0x7f08600ceb00 2024-03-23T15:53:02.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.939+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08601e4770 con 0x7f08600ceb00 2024-03-23T15:53:02.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:02.940+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f086004f530 con 0x7f08600ceb00 2024-03-23T15:53:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.198+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f085002f370 con 0x7f08600ceb00 2024-03-23T15:53:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.198+0000 7f08567fc640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f084406b810 0x7f084406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:03.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.199+0000 7f0865e97640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f084406b810 0x7f084406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:03.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.199+0000 7f0865e97640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f084406b810 0x7f084406dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f085c009b40 tx=0x7f085c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.202+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f085007ae20 con 0x7f08600ceb00 2024-03-23T15:53:03.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.213+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08500770c0 con 0x7f08600ceb00 2024-03-23T15:53:03.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.443+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f086006db50 con 0x7f08600ceb00 2024-03-23T15:53:03.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.461+0000 7f08567fc640 1 -- 172.21.15.129:0/1880212610 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0850034090 con 0x7f08600ceb00 2024-03-23T15:53:03.464 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:03.464 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f084406b810 msgr2=0x7f084406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f084406b810 0x7f084406dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f085c009b40 tx=0x7f085c009190 comp rx=0 tx=0).stop 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 msgr2=0x7f08601e3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 0x7f08601e3640 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0850033000 tx=0x7f0850004730 comp rx=0 tx=0).stop 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 shutdown_connections 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f084406b810 0x7f084406dcd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08600cef40 0x7f0860151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08600ceb00 0x7f08601e3640 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 --2- 172.21.15.129:0/1880212610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08600ccd60 0x7f08601f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:03.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.463+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 >> 172.21.15.129:0/1880212610 conn(0x7f08600b1840 msgr2=0x7f08600ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:03.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.464+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 shutdown_connections 2024-03-23T15:53:03.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:03.464+0000 7f08680fb640 1 -- 172.21.15.129:0/1880212610 wait complete. 2024-03-23T15:53:04.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:04.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:05.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.183+0000 7f2c7ad93640 1 Processor -- start 2024-03-23T15:53:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.183+0000 7f2c7ad93640 1 -- start start 2024-03-23T15:53:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.185+0000 7f2c7ad93640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2c740709e0 0x7f2c740cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.185+0000 7f2c7ad93640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7415c5a0 con 0x7f2c7415ba50 2024-03-23T15:53:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.185+0000 7f2c7ad93640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7406f680 con 0x7f2c7415be30 2024-03-23T15:53:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.185+0000 7f2c7ad93640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2c740705c0 con 0x7f2c740709e0 2024-03-23T15:53:05.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.186+0000 7f2c73fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2c740709e0 0x7f2c740cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:05.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.186+0000 7f2c78b2f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f2c7415ba50 0x7f2c740cc880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36074/0 (socket says 172.21.15.129:36074) 2024-03-23T15:53:05.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.186+0000 7f2c78b2f640 1 -- 172.21.15.129:0/1999786764 learned_addr learned my addr 172.21.15.129:0/1999786764 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:05.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.188+0000 7f2c737fe640 1 -- 172.21.15.129:0/1999786764 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 73355952 0 0) 0x7f2c7415c5a0 con 0x7f2c7415ba50 2024-03-23T15:53:05.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.188+0000 7f2c737fe640 1 -- 172.21.15.129:0/1999786764 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c64003520 con 0x7f2c7415ba50 2024-03-23T15:53:05.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.189+0000 7f2c73fff640 1 -- 172.21.15.129:0/1999786764 >> v1:172.21.15.157:6789/0 conn(0x7f2c7415be30 legacy=0x7f2c740c33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:05.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.189+0000 7f2c73fff640 1 -- 172.21.15.129:0/1999786764 >> v1:172.21.15.129:6789/0 conn(0x7f2c7415ba50 legacy=0x7f2c740cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:05.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.189+0000 7f2c73fff640 1 -- 172.21.15.129:0/1999786764 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7406f940 con 0x7f2c740709e0 2024-03-23T15:53:05.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.193+0000 7f2c73fff640 1 --2- 172.21.15.129:0/1999786764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c740cbac0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c600097a0 tx=0x7f2c6002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4782812af3d3a761 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:05.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.198+0000 7f2c737fe640 1 -- 172.21.15.129:0/1999786764 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60030610 con 0x7f2c740709e0 2024-03-23T15:53:05.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.198+0000 7f2c737fe640 1 -- 172.21.15.129:0/1999786764 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c60030770 con 0x7f2c740709e0 2024-03-23T15:53:05.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.198+0000 7f2c737fe640 1 -- 172.21.15.129:0/1999786764 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60030b00 con 0x7f2c740709e0 2024-03-23T15:53:05.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 -- 172.21.15.129:0/1999786764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 msgr2=0x7f2c740cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:05.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/1999786764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c740cbac0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c600097a0 tx=0x7f2c6002b960 comp rx=0 tx=0).stop 2024-03-23T15:53:05.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 -- 172.21.15.129:0/1999786764 shutdown_connections 2024-03-23T15:53:05.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/1999786764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c740cbac0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 -- 172.21.15.129:0/1999786764 >> 172.21.15.129:0/1999786764 conn(0x7f2c74071300 msgr2=0x7f2c740716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:05.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 -- 172.21.15.129:0/1999786764 shutdown_connections 2024-03-23T15:53:05.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.199+0000 7f2c7ad93640 1 -- 172.21.15.129:0/1999786764 wait complete. 2024-03-23T15:53:05.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.201+0000 7f2c7ad93640 1 Processor -- start 2024-03-23T15:53:05.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.201+0000 7f2c7ad93640 1 -- start start 2024-03-23T15:53:05.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c7ad93640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c741edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:05.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c7ad93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c7415ba50 0x7f2c741560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:05.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c7ad93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c7415be30 0x7f2c741f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:05.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c7ad93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741f18d0 con 0x7f2c7415be30 2024-03-23T15:53:05.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c7ad93640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2c74159140 con 0x7f2c740709e0 2024-03-23T15:53:05.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c7ad93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c741eee80 con 0x7f2c7415ba50 2024-03-23T15:53:05.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c79330640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c7415be30 0x7f2c741f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:05.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c78b2f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c741edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:05.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.203+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c7415ba50 0x7f2c741560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:05.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.204+0000 7f2c78b2f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c741edf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:57076/0 (socket says 172.21.15.129:57076) 2024-03-23T15:53:05.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.204+0000 7f2c78b2f640 1 -- 172.21.15.129:0/3885360591 learned_addr learned my addr 172.21.15.129:0/3885360591 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:05.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.208+0000 7f2c73fff640 1 -- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 msgr2=0x7f2c741edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:05.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.208+0000 7f2c73fff640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c741edf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.208+0000 7f2c73fff640 1 -- 172.21.15.129:0/3885360591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c7415be30 msgr2=0x7f2c741f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:05.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.208+0000 7f2c73fff640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c7415be30 0x7f2c741f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.208+0000 7f2c73fff640 1 -- 172.21.15.129:0/3885360591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c60009400 con 0x7f2c7415ba50 2024-03-23T15:53:05.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.211+0000 7f2c73fff640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c7415ba50 0x7f2c741560a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2c60033000 tx=0x7f2c60004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:05.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.215+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6003e040 con 0x7f2c7415ba50 2024-03-23T15:53:05.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.215+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c60039cd0 con 0x7f2c7415ba50 2024-03-23T15:53:05.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.215+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60044630 con 0x7f2c7415ba50 2024-03-23T15:53:05.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.215+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c741ee5e0 con 0x7f2c7415ba50 2024-03-23T15:53:05.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.215+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c741eeba0 con 0x7f2c7415ba50 2024-03-23T15:53:05.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.216+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c7415c270 con 0x7f2c7415ba50 2024-03-23T15:53:05.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.235+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2c60044790 con 0x7f2c7415ba50 2024-03-23T15:53:05.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.236+0000 7f2c717fa640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c5806b890 0x7f2c5806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:05.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.236+0000 7f2c78b2f640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c5806b890 0x7f2c5806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:05.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.236+0000 7f2c78b2f640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c5806b890 0x7f2c5806dd50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005880 tx=0x7f2c68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:05.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.237+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2c60039e40 con 0x7f2c7415ba50 2024-03-23T15:53:05.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.487+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c60077510 con 0x7f2c7415ba50 2024-03-23T15:53:05.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.710+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c74068a20 con 0x7f2c7415ba50 2024-03-23T15:53:05.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.722+0000 7f2c717fa640 1 -- 172.21.15.129:0/3885360591 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2c60034090 con 0x7f2c7415ba50 2024-03-23T15:53:05.726 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:05.726 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:05.726 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:05.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c5806b890 msgr2=0x7f2c5806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:05.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c5806b890 0x7f2c5806dd50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005880 tx=0x7f2c68009040 comp rx=0 tx=0).stop 2024-03-23T15:53:05.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c7415ba50 msgr2=0x7f2c741560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:05.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c7415ba50 0x7f2c741560a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2c60033000 tx=0x7f2c60004730 comp rx=0 tx=0).stop 2024-03-23T15:53:05.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 shutdown_connections 2024-03-23T15:53:05.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c7415be30 0x7f2c741f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c7415ba50 0x7f2c741560a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c5806b890 0x7f2c5806dd50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 --2- 172.21.15.129:0/3885360591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c740709e0 0x7f2c741edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:05.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 >> 172.21.15.129:0/3885360591 conn(0x7f2c74071300 msgr2=0x7f2c740b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:05.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 shutdown_connections 2024-03-23T15:53:05.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:05.725+0000 7f2c7ad93640 1 -- 172.21.15.129:0/3885360591 wait complete. 2024-03-23T15:53:06.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:06.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:07.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:53:07.179 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:53:07.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.444+0000 7f9a1cad4640 1 Processor -- start 2024-03-23T15:53:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.444+0000 7f9a1cad4640 1 -- start start 2024-03-23T15:53:07.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.445+0000 7f9a1cad4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9a181544b0 0x7f9a1815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:07.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.445+0000 7f9a1cad4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a180cbcb0 con 0x7f9a180caa00 2024-03-23T15:53:07.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.445+0000 7f9a1cad4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a1815b280 con 0x7f9a180c6000 2024-03-23T15:53:07.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.445+0000 7f9a1cad4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9a180c5be0 con 0x7f9a181544b0 2024-03-23T15:53:07.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.446+0000 7f9a1659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9a181544b0 0x7f9a1815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:07.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.446+0000 7f9a15d9b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9a180caa00 0x7f9a1806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36086/0 (socket says 172.21.15.129:36086) 2024-03-23T15:53:07.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.446+0000 7f9a15d9b640 1 -- 172.21.15.129:0/2292625991 learned_addr learned my addr 172.21.15.129:0/2292625991 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:07.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.448+0000 7f9a1559a640 1 -- 172.21.15.129:0/2292625991 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669134425 0 0) 0x7f9a180cbcb0 con 0x7f9a180caa00 2024-03-23T15:53:07.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.448+0000 7f9a1559a640 1 -- 172.21.15.129:0/2292625991 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a00003520 con 0x7f9a180caa00 2024-03-23T15:53:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.450+0000 7f9a1659c640 1 -- 172.21.15.129:0/2292625991 >> v1:172.21.15.157:6789/0 conn(0x7f9a180c6000 legacy=0x7f9a18164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.450+0000 7f9a1659c640 1 -- 172.21.15.129:0/2292625991 >> v1:172.21.15.129:6789/0 conn(0x7f9a180caa00 legacy=0x7f9a1806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:07.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.450+0000 7f9a1659c640 1 -- 172.21.15.129:0/2292625991 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1815a9e0 con 0x7f9a181544b0 2024-03-23T15:53:07.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.456+0000 7f9a1659c640 1 --2- 172.21.15.129:0/2292625991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a1815a2e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9a080097a0 tx=0x7f9a0802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0a1faa3b2178441 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:07.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.462+0000 7f9a1559a640 1 -- 172.21.15.129:0/2292625991 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0802f810 con 0x7f9a181544b0 2024-03-23T15:53:07.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.462+0000 7f9a1559a640 1 -- 172.21.15.129:0/2292625991 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0802f970 con 0x7f9a181544b0 2024-03-23T15:53:07.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.462+0000 7f9a1559a640 1 -- 172.21.15.129:0/2292625991 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0802fd00 con 0x7f9a181544b0 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 -- 172.21.15.129:0/2292625991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 msgr2=0x7f9a1815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/2292625991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a1815a2e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9a080097a0 tx=0x7f9a0802b9f0 comp rx=0 tx=0).stop 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 -- 172.21.15.129:0/2292625991 shutdown_connections 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/2292625991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a1815a2e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 -- 172.21.15.129:0/2292625991 >> 172.21.15.129:0/2292625991 conn(0x7f9a18061b50 msgr2=0x7f9a18061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 -- 172.21.15.129:0/2292625991 shutdown_connections 2024-03-23T15:53:07.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.463+0000 7f9a1cad4640 1 -- 172.21.15.129:0/2292625991 wait complete. 2024-03-23T15:53:07.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.465+0000 7f9a1cad4640 1 Processor -- start 2024-03-23T15:53:07.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.465+0000 7f9a1cad4640 1 -- start start 2024-03-23T15:53:07.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.466+0000 7f9a1cad4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a180c6000 0x7f9a180c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:07.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.466+0000 7f9a1cad4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a180caa00 0x7f9a1814d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:07.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.466+0000 7f9a1cad4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a180cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:07.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.466+0000 7f9a1cad4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18157770 con 0x7f9a180caa00 2024-03-23T15:53:07.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.466+0000 7f9a1cad4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a180c75a0 con 0x7f9a181544b0 2024-03-23T15:53:07.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.466+0000 7f9a1cad4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a18156e80 con 0x7f9a180c6000 2024-03-23T15:53:07.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.467+0000 7f9a16d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a180cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:07.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.467+0000 7f9a15d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a180caa00 0x7f9a1814d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:07.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.467+0000 7f9a1659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a180c6000 0x7f9a180c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:07.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.467+0000 7f9a16d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a180cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:57094/0 (socket says 172.21.15.129:57094) 2024-03-23T15:53:07.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.467+0000 7f9a16d9d640 1 -- 172.21.15.129:0/624066918 learned_addr learned my addr 172.21.15.129:0/624066918 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.472+0000 7f9a16d9d640 1 -- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a180c6000 msgr2=0x7f9a180c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.472+0000 7f9a16d9d640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a180c6000 0x7f9a180c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.472+0000 7f9a16d9d640 1 -- 172.21.15.129:0/624066918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a180caa00 msgr2=0x7f9a1814d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.472+0000 7f9a16d9d640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a180caa00 0x7f9a1814d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.472+0000 7f9a16d9d640 1 -- 172.21.15.129:0/624066918 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a08009400 con 0x7f9a181544b0 2024-03-23T15:53:07.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.475+0000 7f9a16d9d640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a180cef80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c7a0 tx=0x7f9a1000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:07.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.477+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10003b70 con 0x7f9a181544b0 2024-03-23T15:53:07.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.477+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a180c9ec0 con 0x7f9a181544b0 2024-03-23T15:53:07.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.477+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a18157400 con 0x7f9a181544b0 2024-03-23T15:53:07.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.479+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a181578e0 con 0x7f9a181544b0 2024-03-23T15:53:07.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.480+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a10001830 con 0x7f9a181544b0 2024-03-23T15:53:07.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.480+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000a850 con 0x7f9a181544b0 2024-03-23T15:53:07.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.765+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9a1000aa70 con 0x7f9a181544b0 2024-03-23T15:53:07.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.766+0000 7f9a06ffd640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f99f806b810 0x7f99f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:07.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.766+0000 7f9a1659c640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f99f806b810 0x7f99f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:07.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.766+0000 7f9a1659c640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f99f806b810 0x7f99f806dcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9a08036000 tx=0x7f9a08008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:07.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.772+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9a100592b0 con 0x7f9a181544b0 2024-03-23T15:53:07.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:07.783+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9a10055550 con 0x7f9a181544b0 2024-03-23T15:53:08.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.038+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a18157590 con 0x7f9a181544b0 2024-03-23T15:53:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.052+0000 7f9a06ffd640 1 -- 172.21.15.129:0/624066918 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9a10058c10 con 0x7f9a181544b0 2024-03-23T15:53:08.056 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:08.056 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:08.056 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f99f806b810 msgr2=0x7f99f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f99f806b810 0x7f99f806dcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9a08036000 tx=0x7f9a08008030 comp rx=0 tx=0).stop 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 msgr2=0x7f9a180cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a180cef80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9a1000c7a0 tx=0x7f9a1000cc60 comp rx=0 tx=0).stop 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 shutdown_connections 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f99f806b810 0x7f99f806dcd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a181544b0 0x7f9a180cef80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a180caa00 0x7f9a1814d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 --2- 172.21.15.129:0/624066918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a180c6000 0x7f9a180c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:08.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 >> 172.21.15.129:0/624066918 conn(0x7f9a18061b50 msgr2=0x7f9a180c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:08.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 shutdown_connections 2024-03-23T15:53:08.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:08.055+0000 7f9a1cad4640 1 -- 172.21.15.129:0/624066918 wait complete. 2024-03-23T15:53:08.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:08.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:09.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:09.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.775+0000 7f077f884640 1 Processor -- start 2024-03-23T15:53:09.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.775+0000 7f077f884640 1 -- start start 2024-03-23T15:53:09.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.776+0000 7f077f884640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f077815e4b0 0x7f077806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.776+0000 7f077f884640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780cb810 con 0x7f077815d9c0 2024-03-23T15:53:09.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.776+0000 7f077f884640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780cc3b0 con 0x7f07780ca4a0 2024-03-23T15:53:09.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.776+0000 7f077f884640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f07780cac00 con 0x7f077815e4b0 2024-03-23T15:53:09.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.777+0000 7f077d620640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f077815d9c0 0x7f07780cbca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36096/0 (socket says 172.21.15.129:36096) 2024-03-23T15:53:09.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.777+0000 7f077d620640 1 -- 172.21.15.129:0/1838289138 learned_addr learned my addr 172.21.15.129:0/1838289138 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:09.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.777+0000 7f077de21640 1 --2- 172.21.15.129:0/1838289138 >> v2:172.21.15.157:3300/0 conn(0x7f077815e4b0 0x7f077806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:09.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.780+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4032684488 0 0) 0x7f07780cb810 con 0x7f077815d9c0 2024-03-23T15:53:09.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.780+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0764003520 con 0x7f077815d9c0 2024-03-23T15:53:09.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1464898408 0 0) 0x7f0764003520 con 0x7f077815d9c0 2024-03-23T15:53:09.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 >> v2:172.21.15.157:3300/0 conn(0x7f077815e4b0 msgr2=0x7f077806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:09.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 --2- 172.21.15.129:0/1838289138 >> v2:172.21.15.157:3300/0 conn(0x7f077815e4b0 0x7f077806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 >> v1:172.21.15.157:6789/0 conn(0x7f07780ca4a0 legacy=0x7f077815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:09.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07780caee0 con 0x7f077815d9c0 2024-03-23T15:53:09.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f0774001670 con 0x7f077815d9c0 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 >> v1:172.21.15.129:6789/0 conn(0x7f077815d9c0 legacy=0x7f07780cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.783+0000 7f076ffff640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07640039c0 0x7f0764003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.784+0000 7f076ffff640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07640047c0 0x7f0764004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.784+0000 7f076ffff640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0764005180 0x7f076400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.784+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07780cac00 con 0x7f07640039c0 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.784+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f076400c080 con 0x7f07640047c0 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.784+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f076400c240 con 0x7f0764005180 2024-03-23T15:53:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.784+0000 7f077de21640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0764005180 0x7f076400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:09.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.787+0000 7f077d620640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07640039c0 0x7f0764003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:09.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.788+0000 7f077de21640 1 -- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07640047c0 msgr2=0x7f0764004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:09.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.788+0000 7f077de21640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07640047c0 0x7f0764004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.788+0000 7f077de21640 1 -- 172.21.15.129:0/1838289138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07640039c0 msgr2=0x7f0764003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:09.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.788+0000 7f077de21640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07640039c0 0x7f0764003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.788+0000 7f077de21640 1 -- 172.21.15.129:0/1838289138 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0764003520 con 0x7f0764005180 2024-03-23T15:53:09.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.789+0000 7f077de21640 1 -- 172.21.15.129:0/1838289138 reap_dead start 2024-03-23T15:53:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.791+0000 7f077de21640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0764005180 0x7f076400b980 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0770009f90 tx=0x7f07640057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a58efb82096687 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:09.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.793+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077002cc20 con 0x7f0764005180 2024-03-23T15:53:09.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.795+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077000f040 con 0x7f0764005180 2024-03-23T15:53:09.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.795+0000 7f076ffff640 1 -- 172.21.15.129:0/1838289138 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770034440 con 0x7f0764005180 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.795+0000 7f077f884640 1 -- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0764005180 msgr2=0x7f076400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.795+0000 7f077f884640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0764005180 0x7f076400b980 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0770009f90 tx=0x7f07640057f0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.795+0000 7f077f884640 1 -- 172.21.15.129:0/1838289138 shutdown_connections 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.795+0000 7f077f884640 1 --2- 172.21.15.129:0/1838289138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0764005180 0x7f076400b980 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.796+0000 7f077f884640 1 -- 172.21.15.129:0/1838289138 >> 172.21.15.129:0/1838289138 conn(0x7f07780cfb20 msgr2=0x7f07780cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.796+0000 7f077f884640 1 -- 172.21.15.129:0/1838289138 shutdown_connections 2024-03-23T15:53:09.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.796+0000 7f077f884640 1 -- 172.21.15.129:0/1838289138 wait complete. 2024-03-23T15:53:09.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 Processor -- start 2024-03-23T15:53:09.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 -- start start 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0764005180 0x7f077815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 0x7f07781511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f077814f8f0 0x7f077814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f077814f4d0 con 0x7f0778150d70 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07781516f0 con 0x7f0764005180 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.797+0000 7f077f884640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f07781502b0 con 0x7f077814f8f0 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.798+0000 7f077ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 0x7f07781511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.798+0000 7f077d620640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0764005180 0x7f077815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.798+0000 7f077de21640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f077814f8f0 0x7f077814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:09.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.798+0000 7f077ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 0x7f07781511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41812/0 (socket says 172.21.15.129:41812) 2024-03-23T15:53:09.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.798+0000 7f077ce1f640 1 -- 172.21.15.129:0/2047028063 learned_addr learned my addr 172.21.15.129:0/2047028063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:09.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.803+0000 7f077ce1f640 1 -- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f077814f8f0 msgr2=0x7f077814fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:09.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.803+0000 7f077ce1f640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f077814f8f0 0x7f077814fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.803+0000 7f077ce1f640 1 -- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0764005180 msgr2=0x7f077815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:09.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.803+0000 7f077ce1f640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0764005180 0x7f077815aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:09.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.803+0000 7f077ce1f640 1 -- 172.21.15.129:0/2047028063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077814ee50 con 0x7f0778150d70 2024-03-23T15:53:09.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.806+0000 7f077ce1f640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 0x7f07781511b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f076800c530 tx=0x7f076800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.811+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680052d0 con 0x7f0778150d70 2024-03-23T15:53:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.811+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768005450 con 0x7f0778150d70 2024-03-23T15:53:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.811+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07781f0a30 con 0x7f0778150d70 2024-03-23T15:53:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.811+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768012870 con 0x7f0778150d70 2024-03-23T15:53:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.811+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07781f0d10 con 0x7f0778150d70 2024-03-23T15:53:09.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:09.812+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f077814f740 con 0x7f0778150d70 2024-03-23T15:53:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.083+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f07680129d0 con 0x7f0778150d70 2024-03-23T15:53:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.084+0000 7f076dffb640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f075c06b810 0x7f075c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.084+0000 7f077d620640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f075c06b810 0x7f075c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.084+0000 7f077d620640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f075c06b810 0x7f075c06dcd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0774000c00 tx=0x7f0774000ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.100+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f07680595b0 con 0x7f0778150d70 2024-03-23T15:53:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.100+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0768055150 con 0x7f0778150d70 2024-03-23T15:53:10.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.320+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f077806dda0 con 0x7f0778150d70 2024-03-23T15:53:10.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.335+0000 7f076dffb640 1 -- 172.21.15.129:0/2047028063 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0768058810 con 0x7f0778150d70 2024-03-23T15:53:10.339 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:10.339 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:10.339 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f075c06b810 msgr2=0x7f075c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f075c06b810 0x7f075c06dcd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0774000c00 tx=0x7f0774000ed0 comp rx=0 tx=0).stop 2024-03-23T15:53:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 msgr2=0x7f07781511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 0x7f07781511b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f076800c530 tx=0x7f076800c930 comp rx=0 tx=0).stop 2024-03-23T15:53:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 shutdown_connections 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f077814f8f0 0x7f077814fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0778150d70 0x7f07781511b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f075c06b810 0x7f075c06dcd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 --2- 172.21.15.129:0/2047028063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0764005180 0x7f077815aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 >> 172.21.15.129:0/2047028063 conn(0x7f07780cfb20 msgr2=0x7f077815c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 shutdown_connections 2024-03-23T15:53:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:10.338+0000 7f077f884640 1 -- 172.21.15.129:0/2047028063 wait complete. 2024-03-23T15:53:10.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:10.957 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:11.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.057+0000 7f1204b1e640 1 Processor -- start 2024-03-23T15:53:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.057+0000 7f1204b1e640 1 -- start start 2024-03-23T15:53:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.058+0000 7f1204b1e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120006c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.058+0000 7f1204b1e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f120006c650 0x7f12000c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.058+0000 7f1204b1e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f120015d880 con 0x7f1200154270 2024-03-23T15:53:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.058+0000 7f1204b1e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f120006ca60 con 0x7f120006c650 2024-03-23T15:53:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.058+0000 7f1204b1e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12000ca9f0 con 0x7f12000ca580 2024-03-23T15:53:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.059+0000 7f11fdd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120006c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.059+0000 7f11fe59c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1200154270 0x7f120015ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42746/0 (socket says 172.21.15.129:42746) 2024-03-23T15:53:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.059+0000 7f11fed9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f120006c650 0x7f12000c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.059+0000 7f11fe59c640 1 -- 172.21.15.129:0/3724633377 learned_addr learned my addr 172.21.15.129:0/3724633377 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.062+0000 7f11fdd9b640 1 -- 172.21.15.129:0/3724633377 >> v2:172.21.15.129:3300/0 conn(0x7f120006c650 msgr2=0x7f12000c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.062+0000 7f11fdd9b640 1 --2- 172.21.15.129:0/3724633377 >> v2:172.21.15.129:3300/0 conn(0x7f120006c650 0x7f12000c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.062+0000 7f11fdd9b640 1 -- 172.21.15.129:0/3724633377 >> v1:172.21.15.157:6789/0 conn(0x7f1200154270 legacy=0x7f120015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.062+0000 7f11fdd9b640 1 -- 172.21.15.129:0/3724633377 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12000c4ae0 con 0x7f12000ca580 2024-03-23T15:53:12.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.066+0000 7f11fdd9b640 1 --2- 172.21.15.129:0/3724633377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120006c110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11f40097a0 tx=0x7f11f402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f881701211c76c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.069+0000 7f11fd59a640 1 -- 172.21.15.129:0/3724633377 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f402f050 con 0x7f12000ca580 2024-03-23T15:53:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.071+0000 7f11fd59a640 1 -- 172.21.15.129:0/3724633377 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f4031700 con 0x7f12000ca580 2024-03-23T15:53:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.071+0000 7f11fd59a640 1 -- 172.21.15.129:0/3724633377 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4031a90 con 0x7f12000ca580 2024-03-23T15:53:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.072+0000 7f1204b1e640 1 -- 172.21.15.129:0/3724633377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 msgr2=0x7f120006c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.072+0000 7f1204b1e640 1 --2- 172.21.15.129:0/3724633377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120006c110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11f40097a0 tx=0x7f11f402b7a0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.073+0000 7f1204b1e640 1 -- 172.21.15.129:0/3724633377 shutdown_connections 2024-03-23T15:53:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.073+0000 7f1204b1e640 1 --2- 172.21.15.129:0/3724633377 >> v2:172.21.15.129:3300/0 conn(0x7f120006c650 0x7f12000c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.073+0000 7f1204b1e640 1 --2- 172.21.15.129:0/3724633377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120006c110 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.073+0000 7f1204b1e640 1 -- 172.21.15.129:0/3724633377 >> 172.21.15.129:0/3724633377 conn(0x7f120006d290 msgr2=0x7f1200072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.073+0000 7f1204b1e640 1 -- 172.21.15.129:0/3724633377 shutdown_connections 2024-03-23T15:53:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.073+0000 7f1204b1e640 1 -- 172.21.15.129:0/3724633377 wait complete. 2024-03-23T15:53:12.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.075+0000 7f1204b1e640 1 Processor -- start 2024-03-23T15:53:12.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.075+0000 7f1204b1e640 1 -- start start 2024-03-23T15:53:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f1204b1e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f120006c650 0x7f1200146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f1204b1e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f1204b1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 0x7f1200144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f1204b1e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12001e4440 con 0x7f1200154270 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f1204b1e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f12001e4860 con 0x7f120006c650 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f1204b1e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12000cf010 con 0x7f12000ca580 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f11fed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 0x7f1200144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.076+0000 7f11fdd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.077+0000 7f11fe59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f120006c650 0x7f1200146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.077+0000 7f11fed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 0x7f1200144450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44772/0 (socket says 172.21.15.129:44772) 2024-03-23T15:53:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.077+0000 7f11fed9d640 1 -- 172.21.15.129:0/1330674705 learned_addr learned my addr 172.21.15.129:0/1330674705 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.080+0000 7f11fed9d640 1 -- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 msgr2=0x7f120015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.080+0000 7f11fed9d640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.080+0000 7f11fed9d640 1 -- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f120006c650 msgr2=0x7f1200146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.080+0000 7f11fed9d640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f120006c650 0x7f1200146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.080+0000 7f11fed9d640 1 -- 172.21.15.129:0/1330674705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f4009400 con 0x7f1200154270 2024-03-23T15:53:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.083+0000 7f11fed9d640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 0x7f1200144450 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009d30 tx=0x7f11f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.087+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8009ba0 con 0x7f1200154270 2024-03-23T15:53:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.087+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f800f040 con 0x7f1200154270 2024-03-23T15:53:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.087+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12001e4a00 con 0x7f1200154270 2024-03-23T15:53:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.087+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f80146c0 con 0x7f1200154270 2024-03-23T15:53:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.087+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12001e40f0 con 0x7f1200154270 2024-03-23T15:53:12.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.088+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12001e4580 con 0x7f1200154270 2024-03-23T15:53:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.112+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11f8014820 con 0x7f1200154270 2024-03-23T15:53:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.112+0000 7f11eeffd640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f11e006b890 0x7f11e006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.113+0000 7f11fe59c640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f11e006b890 0x7f11e006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.113+0000 7f11fe59c640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f11e006b890 0x7f11e006dd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005880 tx=0x7f11f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:12.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.115+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f11f80599c0 con 0x7f1200154270 2024-03-23T15:53:12.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.361+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11f8055c60 con 0x7f1200154270 2024-03-23T15:53:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.580+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12001e4580 con 0x7f1200154270 2024-03-23T15:53:12.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.627+0000 7f11eeffd640 1 -- 172.21.15.129:0/1330674705 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f12001e4580 con 0x7f1200154270 2024-03-23T15:53:12.631 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:12.631 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:12.631 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f11e006b890 msgr2=0x7f11e006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f11e006b890 0x7f11e006dd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005880 tx=0x7f11f0009040 comp rx=0 tx=0).stop 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 msgr2=0x7f1200144450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 0x7f1200144450 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009d30 tx=0x7f11f800c490 comp rx=0 tx=0).stop 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 shutdown_connections 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1200154270 0x7f1200144450 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12000ca580 0x7f120015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f11e006b890 0x7f11e006dd50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 --2- 172.21.15.129:0/1330674705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f120006c650 0x7f1200146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 >> 172.21.15.129:0/1330674705 conn(0x7f120006d290 msgr2=0x7f12000cc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 shutdown_connections 2024-03-23T15:53:12.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:12.629+0000 7f1204b1e640 1 -- 172.21.15.129:0/1330674705 wait complete. 2024-03-23T15:53:13.247 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:13.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:14.249 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:14.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.348+0000 7f4312083640 1 Processor -- start 2024-03-23T15:53:14.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.348+0000 7f4312083640 1 -- start start 2024-03-23T15:53:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f4312083640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ccd60 0x7f430c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f4312083640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f430c15b290 con 0x7f430c0ceb00 2024-03-23T15:53:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f4312083640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f430c0ce800 con 0x7f430c0cef40 2024-03-23T15:53:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f4312083640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f430c154a70 con 0x7f430c0ccd60 2024-03-23T15:53:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f430affd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f430c0ceb00 0x7f430c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55050/0 (socket says 172.21.15.129:55050) 2024-03-23T15:53:14.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f430affd640 1 -- 172.21.15.129:0/130467913 learned_addr learned my addr 172.21.15.129:0/130467913 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:14.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.349+0000 7f430b7fe640 1 --2- 172.21.15.129:0/130467913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ccd60 0x7f430c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:14.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.352+0000 7f430a7fc640 1 -- 172.21.15.129:0/130467913 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109307659 0 0) 0x7f4300001110 con 0x7f430c0ceb00 2024-03-23T15:53:14.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.352+0000 7f430a7fc640 1 -- 172.21.15.129:0/130467913 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f4003520 con 0x7f430c0ceb00 2024-03-23T15:53:14.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.353+0000 7f430b7fe640 1 -- 172.21.15.129:0/130467913 >> v1:172.21.15.157:6789/0 conn(0x7f430c0cef40 legacy=0x7f430c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.353+0000 7f430b7fe640 1 -- 172.21.15.129:0/130467913 >> v1:172.21.15.129:6789/0 conn(0x7f430c0ceb00 legacy=0x7f430c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.353+0000 7f430b7fe640 1 -- 172.21.15.129:0/130467913 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430c0cd950 con 0x7f430c0ccd60 2024-03-23T15:53:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.355+0000 7f430b7fe640 1 --2- 172.21.15.129:0/130467913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ccd60 0x7f430c15ad50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0097d0 tx=0x7f42fc02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde052aced29e116 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:14.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f430a7fc640 1 -- 172.21.15.129:0/130467913 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc030880 con 0x7f430c0ccd60 2024-03-23T15:53:14.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f430a7fc640 1 -- 172.21.15.129:0/130467913 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42fc0309e0 con 0x7f430c0ccd60 2024-03-23T15:53:14.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f430a7fc640 1 -- 172.21.15.129:0/130467913 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc030d70 con 0x7f430c0ccd60 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 -- 172.21.15.129:0/130467913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ccd60 msgr2=0x7f430c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 --2- 172.21.15.129:0/130467913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ccd60 0x7f430c15ad50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f42fc0097d0 tx=0x7f42fc02b9c0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 -- 172.21.15.129:0/130467913 shutdown_connections 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 --2- 172.21.15.129:0/130467913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ccd60 0x7f430c15ad50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 -- 172.21.15.129:0/130467913 >> 172.21.15.129:0/130467913 conn(0x7f430c0b1840 msgr2=0x7f430c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 -- 172.21.15.129:0/130467913 shutdown_connections 2024-03-23T15:53:14.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.360+0000 7f4312083640 1 -- 172.21.15.129:0/130467913 wait complete. 2024-03-23T15:53:14.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 Processor -- start 2024-03-23T15:53:14.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 -- start start 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f430c0ccd60 0x7f430c15a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ceb00 0x7f430c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 0x7f430c159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f430c152090 con 0x7f430c0cef40 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f430c150ec0 con 0x7f430c0ccd60 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.362+0000 7f4312083640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f430c151130 con 0x7f430c0ceb00 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.363+0000 7f430bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 0x7f430c159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.363+0000 7f430b7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f430c0ccd60 0x7f430c15a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.363+0000 7f430affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ceb00 0x7f430c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.363+0000 7f430bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 0x7f430c159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44778/0 (socket says 172.21.15.129:44778) 2024-03-23T15:53:14.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.363+0000 7f430bfff640 1 -- 172.21.15.129:0/1943143354 learned_addr learned my addr 172.21.15.129:0/1943143354 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.367+0000 7f430bfff640 1 -- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ceb00 msgr2=0x7f430c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.367+0000 7f430bfff640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ceb00 0x7f430c150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.367+0000 7f430bfff640 1 -- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f430c0ccd60 msgr2=0x7f430c15a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.367+0000 7f430bfff640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f430c0ccd60 0x7f430c15a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.367+0000 7f430bfff640 1 -- 172.21.15.129:0/1943143354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc009400 con 0x7f430c0cef40 2024-03-23T15:53:14.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.370+0000 7f430bfff640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 0x7f430c159270 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f430400c6b0 tx=0x7f430400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.375+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304003b70 con 0x7f430c0cef40 2024-03-23T15:53:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.375+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4304003cf0 con 0x7f430c0cef40 2024-03-23T15:53:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.375+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304003ea0 con 0x7f430c0cef40 2024-03-23T15:53:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.375+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f430c151410 con 0x7f430c0cef40 2024-03-23T15:53:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.375+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430c1519d0 con 0x7f430c0cef40 2024-03-23T15:53:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.376+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f430c04f530 con 0x7f430c0cef40 2024-03-23T15:53:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.405+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4304005310 con 0x7f430c0cef40 2024-03-23T15:53:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.405+0000 7f42f3fff640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42f806b840 0x7f42f806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.406+0000 7f430b7fe640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42f806b840 0x7f42f806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:14.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.406+0000 7f430b7fe640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42f806b840 0x7f42f806dd00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42fc036000 tx=0x7f42fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:14.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.411+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4304059100 con 0x7f430c0cef40 2024-03-23T15:53:14.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.657+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f43040553a0 con 0x7f430c0cef40 2024-03-23T15:53:14.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.897+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f430c0686f0 con 0x7f430c0cef40 2024-03-23T15:53:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.969+0000 7f42f3fff640 1 -- 172.21.15.129:0/1943143354 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4304058a60 con 0x7f430c0cef40 2024-03-23T15:53:14.973 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:14.973 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:14.973 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.971+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42f806b840 msgr2=0x7f42f806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.971+0000 7f4312083640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42f806b840 0x7f42f806dd00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42fc036000 tx=0x7f42fc008030 comp rx=0 tx=0).stop 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.971+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 msgr2=0x7f430c159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 0x7f430c159270 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f430400c6b0 tx=0x7f430400cb70 comp rx=0 tx=0).stop 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 shutdown_connections 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f430c0cef40 0x7f430c159270 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f430c0ceb00 0x7f430c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42f806b840 0x7f42f806dd00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 --2- 172.21.15.129:0/1943143354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f430c0ccd60 0x7f430c15a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 >> 172.21.15.129:0/1943143354 conn(0x7f430c0b1840 msgr2=0x7f430c158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 shutdown_connections 2024-03-23T15:53:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:14.972+0000 7f4312083640 1 -- 172.21.15.129:0/1943143354 wait complete. 2024-03-23T15:53:15.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:15.590 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:16.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.691+0000 7f6b5c6e6640 1 Processor -- start 2024-03-23T15:53:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.691+0000 7f6b5c6e6640 1 -- start start 2024-03-23T15:53:16.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.692+0000 7f6b5c6e6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.692+0000 7f6b5c6e6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b540cbcb0 con 0x7f6b540c6000 2024-03-23T15:53:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.692+0000 7f6b5c6e6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b5415b280 con 0x7f6b540caa00 2024-03-23T15:53:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.692+0000 7f6b5c6e6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6b540c5be0 con 0x7f6b541544b0 2024-03-23T15:53:16.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.693+0000 7f6b59c81640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6b540caa00 0x7f6b5406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42756/0 (socket says 172.21.15.129:42756) 2024-03-23T15:53:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.693+0000 7f6b59c81640 1 -- 172.21.15.129:0/3372772344 learned_addr learned my addr 172.21.15.129:0/3372772344 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:16.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.693+0000 7f6b5a482640 1 --2- 172.21.15.129:0/3372772344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:16.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.696+0000 7f6b5a482640 1 -- 172.21.15.129:0/3372772344 >> v1:172.21.15.157:6789/0 conn(0x7f6b540caa00 legacy=0x7f6b5406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:16.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.696+0000 7f6b5a482640 1 -- 172.21.15.129:0/3372772344 >> v1:172.21.15.129:6789/0 conn(0x7f6b540c6000 legacy=0x7f6b54164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:16.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.696+0000 7f6b5a482640 1 -- 172.21.15.129:0/3372772344 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b5415a9e0 con 0x7f6b541544b0 2024-03-23T15:53:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.699+0000 7f6b5a482640 1 --2- 172.21.15.129:0/3372772344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5415a2e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b500097d0 tx=0x7f6b5002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da83c14115fdd211 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:16.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.703+0000 7f6b59480640 1 -- 172.21.15.129:0/3372772344 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50031890 con 0x7f6b541544b0 2024-03-23T15:53:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.703+0000 7f6b59480640 1 -- 172.21.15.129:0/3372772344 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b500319f0 con 0x7f6b541544b0 2024-03-23T15:53:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.703+0000 7f6b59480640 1 -- 172.21.15.129:0/3372772344 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50031d80 con 0x7f6b541544b0 2024-03-23T15:53:16.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/3372772344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 msgr2=0x7f6b5415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/3372772344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5415a2e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b500097d0 tx=0x7f6b5002b930 comp rx=0 tx=0).stop 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/3372772344 shutdown_connections 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/3372772344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5415a2e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/3372772344 >> 172.21.15.129:0/3372772344 conn(0x7f6b54061b50 msgr2=0x7f6b54061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/3372772344 shutdown_connections 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.704+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/3372772344 wait complete. 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.705+0000 7f6b5c6e6640 1 Processor -- start 2024-03-23T15:53:16.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.705+0000 7f6b5c6e6640 1 -- start start 2024-03-23T15:53:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.706+0000 7f6b5c6e6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b540c6000 0x7f6b541f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.706+0000 7f6b5c6e6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 0x7f6b540c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.706+0000 7f6b5c6e6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5414a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.706+0000 7f6b5c6e6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b541f37f0 con 0x7f6b540caa00 2024-03-23T15:53:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.706+0000 7f6b5c6e6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6b540d0710 con 0x7f6b540c6000 2024-03-23T15:53:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.706+0000 7f6b5c6e6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6b540cfe70 con 0x7f6b541544b0 2024-03-23T15:53:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.707+0000 7f6b59c81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 0x7f6b540c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.707+0000 7f6b5a482640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b540c6000 0x7f6b541f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.707+0000 7f6b5ac83640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5414a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.707+0000 7f6b59c81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 0x7f6b540c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44790/0 (socket says 172.21.15.129:44790) 2024-03-23T15:53:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.707+0000 7f6b59c81640 1 -- 172.21.15.129:0/884336951 learned_addr learned my addr 172.21.15.129:0/884336951 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.711+0000 7f6b59c81640 1 -- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 msgr2=0x7f6b5414a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.711+0000 7f6b59c81640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5414a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.711+0000 7f6b59c81640 1 -- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b540c6000 msgr2=0x7f6b541f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.711+0000 7f6b59c81640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b540c6000 0x7f6b541f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.711+0000 7f6b59c81640 1 -- 172.21.15.129:0/884336951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b50009400 con 0x7f6b540caa00 2024-03-23T15:53:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.713+0000 7f6b59c81640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 0x7f6b540c9aa0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400c400 tx=0x7f6b4400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.717+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44005ce0 con 0x7f6b540caa00 2024-03-23T15:53:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.717+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b44005e40 con 0x7f6b540caa00 2024-03-23T15:53:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.717+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b540d0150 con 0x7f6b540caa00 2024-03-23T15:53:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.717+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b440072f0 con 0x7f6b540caa00 2024-03-23T15:53:16.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.717+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b541f3410 con 0x7f6b540caa00 2024-03-23T15:53:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.718+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b541f3930 con 0x7f6b540caa00 2024-03-23T15:53:16.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.979+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6b44007510 con 0x7f6b540caa00 2024-03-23T15:53:16.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.979+0000 7f6b4affd640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b3806b860 0x7f6b3806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:16.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.979+0000 7f6b5a482640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b3806b860 0x7f6b3806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:16.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.980+0000 7f6b5a482640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b3806b860 0x7f6b3806dd20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6b50036000 tx=0x7f6b5003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:16.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.994+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6b44058270 con 0x7f6b540caa00 2024-03-23T15:53:16.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:16.994+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6b44054510 con 0x7f6b540caa00 2024-03-23T15:53:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.246+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b541f3930 con 0x7f6b540caa00 2024-03-23T15:53:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.259+0000 7f6b4affd640 1 -- 172.21.15.129:0/884336951 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6b541f3930 con 0x7f6b540caa00 2024-03-23T15:53:17.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:17.263 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b3806b860 msgr2=0x7f6b3806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b3806b860 0x7f6b3806dd20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6b50036000 tx=0x7f6b5003c040 comp rx=0 tx=0).stop 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 msgr2=0x7f6b540c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 0x7f6b540c9aa0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400c400 tx=0x7f6b4400c8c0 comp rx=0 tx=0).stop 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 shutdown_connections 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b541544b0 0x7f6b5414a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b540caa00 0x7f6b540c9aa0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b3806b860 0x7f6b3806dd20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 --2- 172.21.15.129:0/884336951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b540c6000 0x7f6b541f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 >> 172.21.15.129:0/884336951 conn(0x7f6b54061b50 msgr2=0x7f6b540b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 shutdown_connections 2024-03-23T15:53:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:17.262+0000 7f6b5c6e6640 1 -- 172.21.15.129:0/884336951 wait complete. 2024-03-23T15:53:17.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:17.882 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:18.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.981+0000 7f8fb15c2640 1 Processor -- start 2024-03-23T15:53:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.981+0000 7f8fb15c2640 1 -- start start 2024-03-23T15:53:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.982+0000 7f8fb15c2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8fac0c6000 0x7f8fac0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.982+0000 7f8fb15c2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fac0cbcb0 con 0x7f8fac1544b0 2024-03-23T15:53:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.982+0000 7f8fb15c2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fac15b280 con 0x7f8fac15a2e0 2024-03-23T15:53:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.982+0000 7f8fb15c2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8fac0c5be0 con 0x7f8fac0c6000 2024-03-23T15:53:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.983+0000 7f8faaffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8fac1544b0 0x7f8fac0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55070/0 (socket says 172.21.15.129:55070) 2024-03-23T15:53:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.983+0000 7f8faaffd640 1 -- 172.21.15.129:0/184367686 learned_addr learned my addr 172.21.15.129:0/184367686 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.983+0000 7f8fab7fe640 1 --2- 172.21.15.129:0/184367686 >> v2:172.21.15.157:3300/0 conn(0x7f8fac0c6000 0x7f8fac0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.986+0000 7f8fa9ffb640 1 -- 172.21.15.129:0/184367686 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279562869 0 0) 0x7f8fac0cbcb0 con 0x7f8fac1544b0 2024-03-23T15:53:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.986+0000 7f8fa9ffb640 1 -- 172.21.15.129:0/184367686 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f94003520 con 0x7f8fac1544b0 2024-03-23T15:53:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.987+0000 7f8fab7fe640 1 -- 172.21.15.129:0/184367686 >> v1:172.21.15.157:6789/0 conn(0x7f8fac15a2e0 legacy=0x7f8fac06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.987+0000 7f8fab7fe640 1 -- 172.21.15.129:0/184367686 >> v1:172.21.15.129:6789/0 conn(0x7f8fac1544b0 legacy=0x7f8fac0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.987+0000 7f8fab7fe640 1 -- 172.21.15.129:0/184367686 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac15a9e0 con 0x7f8fac0c6000 2024-03-23T15:53:18.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.994+0000 7f8fab7fe640 1 --2- 172.21.15.129:0/184367686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac0c6000 0x7f8fac0c6470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009d90 tx=0x7f8fa402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723f798a818cbef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:19.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:18.997+0000 7f8fa9ffb640 1 -- 172.21.15.129:0/184367686 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa402cd80 con 0x7f8fac0c6000 2024-03-23T15:53:19.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fa9ffb640 1 -- 172.21.15.129:0/184367686 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa4034440 con 0x7f8fac0c6000 2024-03-23T15:53:19.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fa9ffb640 1 -- 172.21.15.129:0/184367686 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa40347b0 con 0x7f8fac0c6000 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 -- 172.21.15.129:0/184367686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac0c6000 msgr2=0x7f8fac0c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/184367686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac0c6000 0x7f8fac0c6470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009d90 tx=0x7f8fa402bfd0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 -- 172.21.15.129:0/184367686 shutdown_connections 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/184367686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac0c6000 0x7f8fac0c6470 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 -- 172.21.15.129:0/184367686 >> 172.21.15.129:0/184367686 conn(0x7f8fac061b50 msgr2=0x7f8fac061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 -- 172.21.15.129:0/184367686 shutdown_connections 2024-03-23T15:53:19.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.000+0000 7f8fb15c2640 1 -- 172.21.15.129:0/184367686 wait complete. 2024-03-23T15:53:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 Processor -- start 2024-03-23T15:53:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 -- start start 2024-03-23T15:53:19.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fac0c6000 0x7f8fac0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac1544b0 0x7f8fac14f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 0x7f8fac146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac0c8850 con 0x7f8fac15a2e0 2024-03-23T15:53:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac0c74d0 con 0x7f8fac1544b0 2024-03-23T15:53:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.002+0000 7f8fb15c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8fac0c77b0 con 0x7f8fac0c6000 2024-03-23T15:53:19.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.003+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 0x7f8fac146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:19.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.003+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fac0c6000 0x7f8fac0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:19.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.003+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 0x7f8fac146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44792/0 (socket says 172.21.15.129:44792) 2024-03-23T15:53:19.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.003+0000 7f8fab7fe640 1 -- 172.21.15.129:0/2060539622 learned_addr learned my addr 172.21.15.129:0/2060539622 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:19.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.004+0000 7f8faa7fc640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac1544b0 0x7f8fac14f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.007+0000 7f8fab7fe640 1 -- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fac0c6000 msgr2=0x7f8fac0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.007+0000 7f8fab7fe640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fac0c6000 0x7f8fac0c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.007+0000 7f8fab7fe640 1 -- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac1544b0 msgr2=0x7f8fac14f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.007+0000 7f8fab7fe640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac1544b0 0x7f8fac14f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.007+0000 7f8fab7fe640 1 -- 172.21.15.129:0/2060539622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa4009a70 con 0x7f8fac15a2e0 2024-03-23T15:53:19.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.009+0000 7f8fab7fe640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 0x7f8fac146410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009d90 tx=0x7f8fa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:19.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.013+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4004450 con 0x7f8fac15a2e0 2024-03-23T15:53:19.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.013+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa40045d0 con 0x7f8fac15a2e0 2024-03-23T15:53:19.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.013+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa40345b0 con 0x7f8fac15a2e0 2024-03-23T15:53:19.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.013+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac0c7a90 con 0x7f8fac15a2e0 2024-03-23T15:53:19.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.013+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac0c8050 con 0x7f8fac15a2e0 2024-03-23T15:53:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.014+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fac0c57e0 con 0x7f8fac15a2e0 2024-03-23T15:53:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.271+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8fa4034710 con 0x7f8fac15a2e0 2024-03-23T15:53:19.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.271+0000 7f8f937fe640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f9806b810 0x7f8f9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:19.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.272+0000 7f8faaffd640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f9806b810 0x7f8f9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:19.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.272+0000 7f8faaffd640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f9806b810 0x7f8f9806dcd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c0011e0 tx=0x7f8f9c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:19.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.288+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8fa407ba90 con 0x7f8fac15a2e0 2024-03-23T15:53:19.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.289+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8fa4077d30 con 0x7f8fac15a2e0 2024-03-23T15:53:19.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.521+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fac0c57e0 con 0x7f8fac15a2e0 2024-03-23T15:53:19.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.539+0000 7f8f937fe640 1 -- 172.21.15.129:0/2060539622 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8fac0c57e0 con 0x7f8fac15a2e0 2024-03-23T15:53:19.542 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:19.542 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:19.543 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:19.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f9806b810 msgr2=0x7f8f9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:19.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f9806b810 0x7f8f9806dcd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c0011e0 tx=0x7f8f9c007d30 comp rx=0 tx=0).stop 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 msgr2=0x7f8fac146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 0x7f8fac146410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009d90 tx=0x7f8fa4004a40 comp rx=0 tx=0).stop 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 shutdown_connections 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fac15a2e0 0x7f8fac146410 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8f9806b810 0x7f8f9806dcd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fac1544b0 0x7f8fac14f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 --2- 172.21.15.129:0/2060539622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fac0c6000 0x7f8fac0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 >> 172.21.15.129:0/2060539622 conn(0x7f8fac061b50 msgr2=0x7f8fac0b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 shutdown_connections 2024-03-23T15:53:19.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:19.541+0000 7f8fb15c2640 1 -- 172.21.15.129:0/2060539622 wait complete. 2024-03-23T15:53:20.160 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:20.160 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:21.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.259+0000 7f03539f4640 1 Processor -- start 2024-03-23T15:53:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.260+0000 7f03539f4640 1 -- start start 2024-03-23T15:53:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.260+0000 7f03539f4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f034c0c6000 0x7f034c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.260+0000 7f03539f4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c0cbcb0 con 0x7f034c1544b0 2024-03-23T15:53:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.260+0000 7f03539f4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c15b280 con 0x7f034c15a2e0 2024-03-23T15:53:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.260+0000 7f03539f4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f034c0c5be0 con 0x7f034c0c6000 2024-03-23T15:53:21.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.261+0000 7f0350f8f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f034c15a2e0 0x7f034c06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44120/0 (socket says 172.21.15.129:44120) 2024-03-23T15:53:21.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.261+0000 7f0350f8f640 1 -- 172.21.15.129:0/2308073834 learned_addr learned my addr 172.21.15.129:0/2308073834 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:21.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.262+0000 7f0351f91640 1 --2- 172.21.15.129:0/2308073834 >> v2:172.21.15.157:3300/0 conn(0x7f034c0c6000 0x7f034c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:21.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.266+0000 7f0343fff640 1 -- 172.21.15.129:0/2308073834 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158377953 0 0) 0x7f034c0cbcb0 con 0x7f034c1544b0 2024-03-23T15:53:21.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.266+0000 7f0343fff640 1 -- 172.21.15.129:0/2308073834 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0338003520 con 0x7f034c1544b0 2024-03-23T15:53:21.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.268+0000 7f0351f91640 1 -- 172.21.15.129:0/2308073834 >> v1:172.21.15.157:6789/0 conn(0x7f034c15a2e0 legacy=0x7f034c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.268+0000 7f0351f91640 1 -- 172.21.15.129:0/2308073834 >> v1:172.21.15.129:6789/0 conn(0x7f034c1544b0 legacy=0x7f034c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.268+0000 7f0351f91640 1 -- 172.21.15.129:0/2308073834 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c15a9e0 con 0x7f034c0c6000 2024-03-23T15:53:21.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.272+0000 7f0351f91640 1 --2- 172.21.15.129:0/2308073834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c0c6000 0x7f034c0c6470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0344009d90 tx=0x7f034402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac890b2a00732ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:21.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.277+0000 7f0343fff640 1 -- 172.21.15.129:0/2308073834 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034402cd80 con 0x7f034c0c6000 2024-03-23T15:53:21.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.277+0000 7f0343fff640 1 -- 172.21.15.129:0/2308073834 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0344034440 con 0x7f034c0c6000 2024-03-23T15:53:21.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.277+0000 7f0343fff640 1 -- 172.21.15.129:0/2308073834 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03440347d0 con 0x7f034c0c6000 2024-03-23T15:53:21.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 -- 172.21.15.129:0/2308073834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c0c6000 msgr2=0x7f034c0c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 --2- 172.21.15.129:0/2308073834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c0c6000 0x7f034c0c6470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0344009d90 tx=0x7f034402bfd0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 -- 172.21.15.129:0/2308073834 shutdown_connections 2024-03-23T15:53:21.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 --2- 172.21.15.129:0/2308073834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c0c6000 0x7f034c0c6470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 -- 172.21.15.129:0/2308073834 >> 172.21.15.129:0/2308073834 conn(0x7f034c061b50 msgr2=0x7f034c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:21.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 -- 172.21.15.129:0/2308073834 shutdown_connections 2024-03-23T15:53:21.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.278+0000 7f03539f4640 1 -- 172.21.15.129:0/2308073834 wait complete. 2024-03-23T15:53:21.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.280+0000 7f03539f4640 1 Processor -- start 2024-03-23T15:53:21.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.280+0000 7f03539f4640 1 -- start start 2024-03-23T15:53:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f03539f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034c0c6000 0x7f034c0ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f03539f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c1544b0 0x7f034c1474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f03539f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f034c15a2e0 0x7f034c147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f03539f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f034c0ce510 con 0x7f034c0c6000 2024-03-23T15:53:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f03539f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f034c14ee30 con 0x7f034c1544b0 2024-03-23T15:53:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f03539f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f034c14eb70 con 0x7f034c15a2e0 2024-03-23T15:53:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.281+0000 7f0351f91640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f034c15a2e0 0x7f034c147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.282+0000 7f0351790640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034c0c6000 0x7f034c0ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.282+0000 7f0350f8f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c1544b0 0x7f034c1474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.282+0000 7f0351f91640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f034c15a2e0 0x7f034c147a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:47414/0 (socket says 172.21.15.129:47414) 2024-03-23T15:53:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.282+0000 7f0351f91640 1 -- 172.21.15.129:0/2193344786 learned_addr learned my addr 172.21.15.129:0/2193344786 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:21.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.286+0000 7f0351790640 1 -- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f034c15a2e0 msgr2=0x7f034c147a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.286+0000 7f0351790640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f034c15a2e0 0x7f034c147a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.286+0000 7f0351790640 1 -- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c1544b0 msgr2=0x7f034c1474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.286+0000 7f0351790640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c1544b0 0x7f034c1474e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.286+0000 7f0351790640 1 -- 172.21.15.129:0/2193344786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0344009a70 con 0x7f034c0c6000 2024-03-23T15:53:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.288+0000 7f0351790640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034c0c6000 0x7f034c0ceb80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f034800c3c0 tx=0x7f034800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:21.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.292+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03480052d0 con 0x7f034c0c6000 2024-03-23T15:53:21.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.292+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c0cdbe0 con 0x7f034c0c6000 2024-03-23T15:53:21.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.293+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c0ce1a0 con 0x7f034c0c6000 2024-03-23T15:53:21.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.294+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c0cb2d0 con 0x7f034c0c6000 2024-03-23T15:53:21.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.553+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0348007dd0 con 0x7f034c0c6000 2024-03-23T15:53:21.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.553+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03480035f0 con 0x7f034c0c6000 2024-03-23T15:53:21.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.554+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0348003810 con 0x7f034c0c6000 2024-03-23T15:53:21.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.554+0000 7f0341ffb640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f033006b810 0x7f033006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:21.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.554+0000 7f0350f8f640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f033006b810 0x7f033006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:21.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.555+0000 7f0350f8f640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f033006b810 0x7f033006dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f034c1edd90 tx=0x7f033c007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:21.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.568+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f03480586e0 con 0x7f034c0c6000 2024-03-23T15:53:21.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.568+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0348054980 con 0x7f034c0c6000 2024-03-23T15:53:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.792+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f034c075740 con 0x7f034c0c6000 2024-03-23T15:53:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.805+0000 7f0341ffb640 1 -- 172.21.15.129:0/2193344786 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0348021090 con 0x7f034c0c6000 2024-03-23T15:53:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:21.809 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:21.809 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:21.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f033006b810 msgr2=0x7f033006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f033006b810 0x7f033006dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f034c1edd90 tx=0x7f033c007e80 comp rx=0 tx=0).stop 2024-03-23T15:53:21.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034c0c6000 msgr2=0x7f034c0ceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:21.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034c0c6000 0x7f034c0ceb80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f034800c3c0 tx=0x7f034800c880 comp rx=0 tx=0).stop 2024-03-23T15:53:21.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 shutdown_connections 2024-03-23T15:53:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f034c15a2e0 0x7f034c147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f033006b810 0x7f033006dcd0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f034c1edd90 tx=0x7f033c007e80 comp rx=0 tx=0).stop 2024-03-23T15:53:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f034c1544b0 0x7f034c1474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 --2- 172.21.15.129:0/2193344786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034c0c6000 0x7f034c0ceb80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 >> 172.21.15.129:0/2193344786 conn(0x7f034c061b50 msgr2=0x7f034c151e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 shutdown_connections 2024-03-23T15:53:21.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:21.807+0000 7f03539f4640 1 -- 172.21.15.129:0/2193344786 wait complete. 2024-03-23T15:53:22.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:22.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:23.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.528+0000 7ff3036c3640 1 Processor -- start 2024-03-23T15:53:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.528+0000 7ff3036c3640 1 -- start start 2024-03-23T15:53:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.529+0000 7ff3036c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc154340 0x7ff2fc0c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.529+0000 7ff3036c3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2fc15e090 con 0x7ff2fc15e520 2024-03-23T15:53:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.529+0000 7ff3036c3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2fc06ca10 con 0x7ff2fc06bd40 2024-03-23T15:53:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.529+0000 7ff3036c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc15d860 con 0x7ff2fc154340 2024-03-23T15:53:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.529+0000 7ff300c5e640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff2fc15e520 0x7ff2fc15aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:39316/0 (socket says 172.21.15.129:39316) 2024-03-23T15:53:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.529+0000 7ff300c5e640 1 -- 172.21.15.129:0/2377449335 learned_addr learned my addr 172.21.15.129:0/2377449335 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:23.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.530+0000 7ff30145f640 1 --2- 172.21.15.129:0/2377449335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc154340 0x7ff2fc0c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.532+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871374638 0 0) 0x7ff2fc06ca10 con 0x7ff2fc06bd40 2024-03-23T15:53:23.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.532+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e8003520 con 0x7ff2fc06bd40 2024-03-23T15:53:23.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.532+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915365067 0 0) 0x7ff2fc15e090 con 0x7ff2fc15e520 2024-03-23T15:53:23.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.532+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2fc06ca10 con 0x7ff2fc15e520 2024-03-23T15:53:23.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.533+0000 7ff30145f640 1 -- 172.21.15.129:0/2377449335 >> v1:172.21.15.157:6789/0 conn(0x7ff2fc06bd40 legacy=0x7ff2fc0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:23.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.533+0000 7ff30145f640 1 -- 172.21.15.129:0/2377449335 >> v1:172.21.15.129:6789/0 conn(0x7ff2fc15e520 legacy=0x7ff2fc15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:23.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.533+0000 7ff30145f640 1 -- 172.21.15.129:0/2377449335 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc0c74d0 con 0x7ff2fc154340 2024-03-23T15:53:23.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.536+0000 7ff30145f640 1 --2- 172.21.15.129:0/2377449335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc154340 0x7ff2fc0c6f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80097d0 tx=0x7ff2f802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bec654cea2d03bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:23.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8030880 con 0x7ff2fc154340 2024-03-23T15:53:23.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f80309e0 con 0x7ff2fc154340 2024-03-23T15:53:23.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff2f3fff640 1 -- 172.21.15.129:0/2377449335 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8030d70 con 0x7ff2fc154340 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 -- 172.21.15.129:0/2377449335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc154340 msgr2=0x7ff2fc0c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 --2- 172.21.15.129:0/2377449335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc154340 0x7ff2fc0c6f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80097d0 tx=0x7ff2f802b9c0 comp rx=0 tx=0).stop 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 -- 172.21.15.129:0/2377449335 shutdown_connections 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 --2- 172.21.15.129:0/2377449335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc154340 0x7ff2fc0c6f90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 -- 172.21.15.129:0/2377449335 >> 172.21.15.129:0/2377449335 conn(0x7ff2fc071dc0 msgr2=0x7ff2fc0721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 -- 172.21.15.129:0/2377449335 shutdown_connections 2024-03-23T15:53:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.541+0000 7ff3036c3640 1 -- 172.21.15.129:0/2377449335 wait complete. 2024-03-23T15:53:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 Processor -- start 2024-03-23T15:53:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 -- start start 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc06bd40 0x7ff2fc1ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2fc154340 0x7ff2fc149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 0x7ff2fc149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc0ca200 con 0x7ff2fc15e520 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1e9f70 con 0x7ff2fc154340 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.543+0000 7ff3036c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc1f2ec0 con 0x7ff2fc06bd40 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.544+0000 7ff301c60640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 0x7ff2fc149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:23.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.544+0000 7ff300c5e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2fc154340 0x7ff2fc149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:23.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.544+0000 7ff30145f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc06bd40 0x7ff2fc1ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:23.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.544+0000 7ff301c60640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 0x7ff2fc149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34850/0 (socket says 172.21.15.129:34850) 2024-03-23T15:53:23.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.544+0000 7ff301c60640 1 -- 172.21.15.129:0/946960088 learned_addr learned my addr 172.21.15.129:0/946960088 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:23.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.547+0000 7ff301c60640 1 -- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc06bd40 msgr2=0x7ff2fc1ef8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:23.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.547+0000 7ff301c60640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc06bd40 0x7ff2fc1ef8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:23.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.547+0000 7ff301c60640 1 -- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2fc154340 msgr2=0x7ff2fc149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:23.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.547+0000 7ff301c60640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2fc154340 0x7ff2fc149a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:23.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.547+0000 7ff301c60640 1 -- 172.21.15.129:0/946960088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f8009400 con 0x7ff2fc15e520 2024-03-23T15:53:23.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.550+0000 7ff301c60640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 0x7ff2fc149f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4001460 tx=0x7ff2f400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:23.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.554+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4003b70 con 0x7ff2fc15e520 2024-03-23T15:53:23.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.554+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f4003cd0 con 0x7ff2fc15e520 2024-03-23T15:53:23.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.554+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2fc1f3180 con 0x7ff2fc15e520 2024-03-23T15:53:23.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.554+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f400aa30 con 0x7ff2fc15e520 2024-03-23T15:53:23.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.554+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2fc1f3740 con 0x7ff2fc15e520 2024-03-23T15:53:23.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.555+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2fc0ca370 con 0x7ff2fc15e520 2024-03-23T15:53:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.573+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff2f400ab90 con 0x7ff2fc15e520 2024-03-23T15:53:23.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.574+0000 7ff2f1ffb640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2e006b860 0x7ff2e006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:23.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.574+0000 7ff30145f640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2e006b860 0x7ff2e006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:23.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.575+0000 7ff30145f640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2e006b860 0x7ff2e006dd20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8036000 tx=0x7ff2f8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.578+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff2f400ae40 con 0x7ff2fc15e520 2024-03-23T15:53:23.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:23.820+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff2f4058dc0 con 0x7ff2fc15e520 2024-03-23T15:53:24.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.037+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2fc0ca340 con 0x7ff2fc15e520 2024-03-23T15:53:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.049+0000 7ff2f1ffb640 1 -- 172.21.15.129:0/946960088 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff2f4058950 con 0x7ff2fc15e520 2024-03-23T15:53:24.052 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:24.052 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:24.053 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:24.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2e006b860 msgr2=0x7ff2e006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2e006b860 0x7ff2e006dd20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8036000 tx=0x7ff2f8008030 comp rx=0 tx=0).stop 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 msgr2=0x7ff2fc149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 0x7ff2fc149f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4001460 tx=0x7ff2f400cc00 comp rx=0 tx=0).stop 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 shutdown_connections 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2fc15e520 0x7ff2fc149f90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2e006b860 0x7ff2e006dd20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2fc154340 0x7ff2fc149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 --2- 172.21.15.129:0/946960088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2fc06bd40 0x7ff2fc1ef8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 >> 172.21.15.129:0/946960088 conn(0x7ff2fc071dc0 msgr2=0x7ff2fc0721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:24.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 shutdown_connections 2024-03-23T15:53:24.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:24.051+0000 7ff3036c3640 1 -- 172.21.15.129:0/946960088 wait complete. 2024-03-23T15:53:24.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:24.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:25.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:25.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.772+0000 7f17b8702640 1 Processor -- start 2024-03-23T15:53:25.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.772+0000 7f17b8702640 1 -- start start 2024-03-23T15:53:25.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.773+0000 7f17b8702640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17b015c000 0x7f17b015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:25.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.773+0000 7f17b8702640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b015b290 con 0x7f17b00ccd60 2024-03-23T15:53:25.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.773+0000 7f17b8702640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b00ce800 con 0x7f17b00ceb00 2024-03-23T15:53:25.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.773+0000 7f17b8702640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f17b0154a70 con 0x7f17b015c000 2024-03-23T15:53:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.774+0000 7f17b649e640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f17b00ccd60 0x7f17b006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:39320/0 (socket says 172.21.15.129:39320) 2024-03-23T15:53:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.774+0000 7f17b649e640 1 -- 172.21.15.129:0/719109556 learned_addr learned my addr 172.21.15.129:0/719109556 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.774+0000 7f17b6c9f640 1 --2- 172.21.15.129:0/719109556 >> v2:172.21.15.157:3300/0 conn(0x7f17b015c000 0x7f17b015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.777+0000 7f17b549c640 1 -- 172.21.15.129:0/719109556 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345271694 0 0) 0x7f17ac001110 con 0x7f17b00ccd60 2024-03-23T15:53:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.777+0000 7f17b549c640 1 -- 172.21.15.129:0/719109556 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f179c003520 con 0x7f17b00ccd60 2024-03-23T15:53:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.779+0000 7f17b6c9f640 1 -- 172.21.15.129:0/719109556 >> v1:172.21.15.157:6789/0 conn(0x7f17b00ceb00 legacy=0x7f17b015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.779+0000 7f17b6c9f640 1 -- 172.21.15.129:0/719109556 >> v1:172.21.15.129:6789/0 conn(0x7f17b00ccd60 legacy=0x7f17b006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.779+0000 7f17b6c9f640 1 -- 172.21.15.129:0/719109556 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b00cd950 con 0x7f17b015c000 2024-03-23T15:53:25.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.785+0000 7f17b6c9f640 1 --2- 172.21.15.129:0/719109556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b015c000 0x7f17b015ad50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009d90 tx=0x7f17a802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df451643e460b12a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:25.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.791+0000 7f17b549c640 1 -- 172.21.15.129:0/719109556 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a802cd80 con 0x7f17b015c000 2024-03-23T15:53:25.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.791+0000 7f17b549c640 1 -- 172.21.15.129:0/719109556 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a8034440 con 0x7f17b015c000 2024-03-23T15:53:25.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.791+0000 7f17b549c640 1 -- 172.21.15.129:0/719109556 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80347d0 con 0x7f17b015c000 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 -- 172.21.15.129:0/719109556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b015c000 msgr2=0x7f17b015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 --2- 172.21.15.129:0/719109556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b015c000 0x7f17b015ad50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009d90 tx=0x7f17a802bfd0 comp rx=0 tx=0).stop 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 -- 172.21.15.129:0/719109556 shutdown_connections 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 --2- 172.21.15.129:0/719109556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b015c000 0x7f17b015ad50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 -- 172.21.15.129:0/719109556 >> 172.21.15.129:0/719109556 conn(0x7f17b00b1840 msgr2=0x7f17b00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 -- 172.21.15.129:0/719109556 shutdown_connections 2024-03-23T15:53:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.792+0000 7f17b8702640 1 -- 172.21.15.129:0/719109556 wait complete. 2024-03-23T15:53:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.793+0000 7f17b8702640 1 Processor -- start 2024-03-23T15:53:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.793+0000 7f17b8702640 1 -- start start 2024-03-23T15:53:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b8702640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 0x7f17b015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b8702640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17b00ceb00 0x7f17b00c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b8702640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b015c000 0x7f17b00c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b8702640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17b00c6dd0 con 0x7f17b015c000 2024-03-23T15:53:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b8702640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17b00c9d00 con 0x7f17b00ccd60 2024-03-23T15:53:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b8702640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17b00c7040 con 0x7f17b00ceb00 2024-03-23T15:53:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b6c9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b015c000 0x7f17b00c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b649e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 0x7f17b015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b649e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 0x7f17b015df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:34080/0 (socket says 172.21.15.129:34080) 2024-03-23T15:53:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.794+0000 7f17b649e640 1 -- 172.21.15.129:0/475185222 learned_addr learned my addr 172.21.15.129:0/475185222 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:25.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.795+0000 7f17b5c9d640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17b00ceb00 0x7f17b00c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.800+0000 7f17b6c9f640 1 -- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17b00ceb00 msgr2=0x7f17b00c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.800+0000 7f17b6c9f640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17b00ceb00 0x7f17b00c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.800+0000 7f17b6c9f640 1 -- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 msgr2=0x7f17b015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.800+0000 7f17b6c9f640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 0x7f17b015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.800+0000 7f17b6c9f640 1 -- 172.21.15.129:0/475185222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a8009a70 con 0x7f17b015c000 2024-03-23T15:53:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.800+0000 7f17b649e640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 0x7f17b015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:53:25.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.803+0000 7f17b6c9f640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b015c000 0x7f17b00c7a10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17a8033000 tx=0x7f17a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:25.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.807+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a803e410 con 0x7f17b015c000 2024-03-23T15:53:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.807+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a803e5b0 con 0x7f17b015c000 2024-03-23T15:53:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.807+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a803e960 con 0x7f17b015c000 2024-03-23T15:53:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.807+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b01f0b50 con 0x7f17b015c000 2024-03-23T15:53:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.807+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b01e5ee0 con 0x7f17b015c000 2024-03-23T15:53:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:25.808+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b004f530 con 0x7f17b015c000 2024-03-23T15:53:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.072+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f17a803b050 con 0x7f17b015c000 2024-03-23T15:53:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.073+0000 7f17a6ffd640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f179406b860 0x7f179406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.073+0000 7f17b649e640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f179406b860 0x7f179406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.074+0000 7f17b649e640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f179406b860 0x7f179406dd20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17ac005880 tx=0x7f17ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:26.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.090+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f17a807b5c0 con 0x7f17b015c000 2024-03-23T15:53:26.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.090+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f17a8077860 con 0x7f17b015c000 2024-03-23T15:53:26.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.309+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17b00686f0 con 0x7f17b015c000 2024-03-23T15:53:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.323+0000 7f17a6ffd640 1 -- 172.21.15.129:0/475185222 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f17a8039070 con 0x7f17b015c000 2024-03-23T15:53:26.327 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:26.327 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f179406b860 msgr2=0x7f179406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f179406b860 0x7f179406dd20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f17ac005880 tx=0x7f17ac009040 comp rx=0 tx=0).stop 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b015c000 msgr2=0x7f17b00c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b015c000 0x7f17b00c7a10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17a8033000 tx=0x7f17a8004a40 comp rx=0 tx=0).stop 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 shutdown_connections 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17b015c000 0x7f17b00c7a10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17b00ceb00 0x7f17b00c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f179406b860 0x7f179406dd20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 --2- 172.21.15.129:0/475185222 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17b00ccd60 0x7f17b015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 >> 172.21.15.129:0/475185222 conn(0x7f17b00b1840 msgr2=0x7f17b0153360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 shutdown_connections 2024-03-23T15:53:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:26.326+0000 7f17b8702640 1 -- 172.21.15.129:0/475185222 wait complete. 2024-03-23T15:53:26.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:26.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:27.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:28.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 Processor -- start 2024-03-23T15:53:28.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 -- start start 2024-03-23T15:53:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f28a00ccd60 0x7f28a015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28a00cef40 0x7f28a015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a015b290 con 0x7f28a00ceb00 2024-03-23T15:53:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f28a00ce800 con 0x7f28a00cef40 2024-03-23T15:53:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.045+0000 7f28a70cd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f28a0154a70 con 0x7f28a00ccd60 2024-03-23T15:53:28.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.046+0000 7f28a566a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28a00cef40 0x7f28a015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.046+0000 7f28a566a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f28a00cef40 0x7f28a015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34866/0 (socket says 172.21.15.129:34866) 2024-03-23T15:53:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.046+0000 7f28a566a640 1 -- 172.21.15.129:0/582203869 learned_addr learned my addr 172.21.15.129:0/582203869 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:28.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.047+0000 7f28a4e69640 1 --2- 172.21.15.129:0/582203869 >> v2:172.21.15.157:3300/0 conn(0x7f28a00ccd60 0x7f28a015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:28.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.049+0000 7f28a566a640 1 -- 172.21.15.129:0/582203869 >> v2:172.21.15.157:3300/0 conn(0x7f28a00ccd60 msgr2=0x7f28a015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.049+0000 7f28a566a640 1 --2- 172.21.15.129:0/582203869 >> v2:172.21.15.157:3300/0 conn(0x7f28a00ccd60 0x7f28a015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.049+0000 7f28a566a640 1 -- 172.21.15.129:0/582203869 >> v1:172.21.15.157:6789/0 conn(0x7f28a00ceb00 legacy=0x7f28a006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:28.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.049+0000 7f28a566a640 1 -- 172.21.15.129:0/582203869 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a00cd950 con 0x7f28a00cef40 2024-03-23T15:53:28.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.051+0000 7f28a566a640 1 --2- 172.21.15.129:0/582203869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a015b8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f28a008cf40 tx=0x7f289802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7f242638418a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:28.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.055+0000 7f289f7fe640 1 -- 172.21.15.129:0/582203869 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289802cbb0 con 0x7f28a00cef40 2024-03-23T15:53:28.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.055+0000 7f289f7fe640 1 -- 172.21.15.129:0/582203869 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f289802cd10 con 0x7f28a00cef40 2024-03-23T15:53:28.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.055+0000 7f289f7fe640 1 -- 172.21.15.129:0/582203869 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898034610 con 0x7f28a00cef40 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 -- 172.21.15.129:0/582203869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 msgr2=0x7f28a015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 --2- 172.21.15.129:0/582203869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a015b8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f28a008cf40 tx=0x7f289802c010 comp rx=0 tx=0).stop 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 -- 172.21.15.129:0/582203869 shutdown_connections 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 --2- 172.21.15.129:0/582203869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a015b8f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 --2- 172.21.15.129:0/582203869 >> v2:172.21.15.157:3300/0 conn(0x7f28a00ccd60 0x7f28a015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 -- 172.21.15.129:0/582203869 >> 172.21.15.129:0/582203869 conn(0x7f28a00b1840 msgr2=0x7f28a00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 -- 172.21.15.129:0/582203869 shutdown_connections 2024-03-23T15:53:28.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.056+0000 7f28a70cd640 1 -- 172.21.15.129:0/582203869 wait complete. 2024-03-23T15:53:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.057+0000 7f28a70cd640 1 Processor -- start 2024-03-23T15:53:28.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.057+0000 7f28a70cd640 1 -- start start 2024-03-23T15:53:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a70cd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28a00ccd60 0x7f28a015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a70cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28a00ceb00 0x7f28a015c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a70cd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a014b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a70cd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28a015b0a0 con 0x7f28a00cef40 2024-03-23T15:53:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a70cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f28a014c110 con 0x7f28a00ceb00 2024-03-23T15:53:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a70cd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f28a01e4ed0 con 0x7f28a00ccd60 2024-03-23T15:53:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f28a566a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a014b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.058+0000 7f289ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28a00ceb00 0x7f28a015c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.059+0000 7f28a566a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a014b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34870/0 (socket says 172.21.15.129:34870) 2024-03-23T15:53:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.059+0000 7f28a566a640 1 -- 172.21.15.129:0/3093375916 learned_addr learned my addr 172.21.15.129:0/3093375916 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.059+0000 7f28a4e69640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28a00ccd60 0x7f28a015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.062+0000 7f28a566a640 1 -- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28a00ccd60 msgr2=0x7f28a015bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.062+0000 7f28a566a640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28a00ccd60 0x7f28a015bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.062+0000 7f28a566a640 1 -- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28a00ceb00 msgr2=0x7f28a015c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.062+0000 7f28a566a640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28a00ceb00 0x7f28a015c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.062+0000 7f28a566a640 1 -- 172.21.15.129:0/3093375916 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2898009b30 con 0x7f28a00cef40 2024-03-23T15:53:28.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.064+0000 7f28a566a640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a014b0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2898009f40 tx=0x7f2898004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.069+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289803e450 con 0x7f28a00cef40 2024-03-23T15:53:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.069+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f289803e5d0 con 0x7f28a00cef40 2024-03-23T15:53:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.069+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289803e980 con 0x7f28a00cef40 2024-03-23T15:53:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.069+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a01e4070 con 0x7f28a00cef40 2024-03-23T15:53:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.069+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a01e4630 con 0x7f28a00cef40 2024-03-23T15:53:28.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.070+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a004f530 con 0x7f28a00cef40 2024-03-23T15:53:28.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.089+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f289803eae0 con 0x7f28a00cef40 2024-03-23T15:53:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.090+0000 7f289d7fa640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f288406b840 0x7f288406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.090+0000 7f28a4e69640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f288406b840 0x7f288406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.090+0000 7f28a4e69640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f288406b840 0x7f288406dd00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28940079a0 tx=0x7f2894008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:28.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.093+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f289807bab0 con 0x7f28a00cef40 2024-03-23T15:53:28.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.336+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2898077d50 con 0x7f28a00cef40 2024-03-23T15:53:28.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.605+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28a00686f0 con 0x7f28a00cef40 2024-03-23T15:53:28.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.638+0000 7f289d7fa640 1 -- 172.21.15.129:0/3093375916 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2898043020 con 0x7f28a00cef40 2024-03-23T15:53:28.642 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:28.642 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:28.643 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f288406b840 msgr2=0x7f288406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f288406b840 0x7f288406dd00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28940079a0 tx=0x7f2894008040 comp rx=0 tx=0).stop 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 msgr2=0x7f28a014b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a014b0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2898009f40 tx=0x7f2898004a40 comp rx=0 tx=0).stop 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 shutdown_connections 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28a00cef40 0x7f28a014b0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f288406b840 0x7f288406dd00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28a00ceb00 0x7f28a015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 --2- 172.21.15.129:0/3093375916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28a00ccd60 0x7f28a015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:28.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 >> 172.21.15.129:0/3093375916 conn(0x7f28a00b1840 msgr2=0x7f28a00ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:28.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 shutdown_connections 2024-03-23T15:53:28.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:28.641+0000 7f28a70cd640 1 -- 172.21.15.129:0/3093375916 wait complete. 2024-03-23T15:53:29.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:29.259 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:30.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.359+0000 7ffb2fed4640 1 Processor -- start 2024-03-23T15:53:30.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.360+0000 7ffb2fed4640 1 -- start start 2024-03-23T15:53:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.361+0000 7ffb2fed4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffb281544b0 0x7ffb2815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.361+0000 7ffb2fed4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb280c6000 0x7ffb280c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.361+0000 7ffb2fed4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb280cbcb0 con 0x7ffb280caa00 2024-03-23T15:53:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.361+0000 7ffb2fed4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ffb280c5be0 con 0x7ffb280c6000 2024-03-23T15:53:30.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.361+0000 7ffb2fed4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffb280c69b0 con 0x7ffb281544b0 2024-03-23T15:53:30.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.362+0000 7ffb2e471640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffb280c6000 0x7ffb280c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:30.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.362+0000 7ffb2d46f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ffb280caa00 0x7ffb2806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44138/0 (socket says 172.21.15.129:44138) 2024-03-23T15:53:30.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.362+0000 7ffb2d46f640 1 -- 172.21.15.129:0/3267988322 learned_addr learned my addr 172.21.15.129:0/3267988322 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:30.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.362+0000 7ffb2dc70640 1 --2- 172.21.15.129:0/3267988322 >> v2:172.21.15.157:3300/0 conn(0x7ffb281544b0 0x7ffb2815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:30.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.365+0000 7ffb2e471640 1 -- 172.21.15.129:0/3267988322 >> v2:172.21.15.157:3300/0 conn(0x7ffb281544b0 msgr2=0x7ffb2815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:30.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.365+0000 7ffb2e471640 1 --2- 172.21.15.129:0/3267988322 >> v2:172.21.15.157:3300/0 conn(0x7ffb281544b0 0x7ffb2815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.365+0000 7ffb2e471640 1 -- 172.21.15.129:0/3267988322 >> v1:172.21.15.157:6789/0 conn(0x7ffb280caa00 legacy=0x7ffb2806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:30.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.365+0000 7ffb2e471640 1 -- 172.21.15.129:0/3267988322 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2815b280 con 0x7ffb280c6000 2024-03-23T15:53:30.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.368+0000 7ffb2e471640 1 --2- 172.21.15.129:0/3267988322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280c6000 0x7ffb280c6470 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffb2806cd70 tx=0x7ffb2002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53adb9421605af82 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:30.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.373+0000 7ffb2cc6e640 1 -- 172.21.15.129:0/3267988322 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2002cd00 con 0x7ffb280c6000 2024-03-23T15:53:30.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.373+0000 7ffb2cc6e640 1 -- 172.21.15.129:0/3267988322 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2002ce60 con 0x7ffb280c6000 2024-03-23T15:53:30.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.373+0000 7ffb2cc6e640 1 -- 172.21.15.129:0/3267988322 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20034660 con 0x7ffb280c6000 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 -- 172.21.15.129:0/3267988322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280c6000 msgr2=0x7ffb280c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/3267988322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280c6000 0x7ffb280c6470 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffb2806cd70 tx=0x7ffb2002bf50 comp rx=0 tx=0).stop 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 -- 172.21.15.129:0/3267988322 shutdown_connections 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/3267988322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280c6000 0x7ffb280c6470 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/3267988322 >> v2:172.21.15.157:3300/0 conn(0x7ffb281544b0 0x7ffb2815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 -- 172.21.15.129:0/3267988322 >> 172.21.15.129:0/3267988322 conn(0x7ffb28061b50 msgr2=0x7ffb28061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 -- 172.21.15.129:0/3267988322 shutdown_connections 2024-03-23T15:53:30.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.374+0000 7ffb2fed4640 1 -- 172.21.15.129:0/3267988322 wait complete. 2024-03-23T15:53:30.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.375+0000 7ffb2fed4640 1 Processor -- start 2024-03-23T15:53:30.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.375+0000 7ffb2fed4640 1 -- start start 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2fed4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb280c6000 0x7ffb28147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2fed4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280caa00 0x7ffb281f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2fed4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb281544b0 0x7ffb281f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2fed4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffb280cef10 con 0x7ffb280caa00 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2fed4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb280cda60 con 0x7ffb280c6000 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2fed4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffb280d0560 con 0x7ffb281544b0 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2d46f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280caa00 0x7ffb281f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2e471640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb281544b0 0x7ffb281f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2d46f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280caa00 0x7ffb281f1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34884/0 (socket says 172.21.15.129:34884) 2024-03-23T15:53:30.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.376+0000 7ffb2d46f640 1 -- 172.21.15.129:0/4191876713 learned_addr learned my addr 172.21.15.129:0/4191876713 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:30.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.380+0000 7ffb2e471640 1 -- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb280c6000 msgr2=0x7ffb28147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:30.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.380+0000 7ffb2e471640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb280c6000 0x7ffb28147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.380+0000 7ffb2e471640 1 -- 172.21.15.129:0/4191876713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280caa00 msgr2=0x7ffb281f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:30.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.380+0000 7ffb2e471640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280caa00 0x7ffb281f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.380+0000 7ffb2e471640 1 -- 172.21.15.129:0/4191876713 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb20009a70 con 0x7ffb281544b0 2024-03-23T15:53:30.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.383+0000 7ffb2e471640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb281544b0 0x7ffb281f1ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb20033000 tx=0x7ffb20004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:30.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.388+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2003d600 con 0x7ffb281544b0 2024-03-23T15:53:30.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.388+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2003d780 con 0x7ffb281544b0 2024-03-23T15:53:30.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.388+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2003db30 con 0x7ffb281544b0 2024-03-23T15:53:30.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.388+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb280d0840 con 0x7ffb281544b0 2024-03-23T15:53:30.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.388+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb280cdf90 con 0x7ffb281544b0 2024-03-23T15:53:30.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.389+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb280ce120 con 0x7ffb281544b0 2024-03-23T15:53:30.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.661+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffb2003b040 con 0x7ffb281544b0 2024-03-23T15:53:30.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.662+0000 7ffb1e7fc640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffb0c06b890 0x7ffb0c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:30.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.662+0000 7ffb2dc70640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffb0c06b890 0x7ffb0c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:30.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.663+0000 7ffb2dc70640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffb0c06b890 0x7ffb0c06dd50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb24006a90 tx=0x7ffb24006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:30.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.668+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ffb2007ad60 con 0x7ffb281544b0 2024-03-23T15:53:30.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.687+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffb20077000 con 0x7ffb281544b0 2024-03-23T15:53:30.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.908+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb28075740 con 0x7ffb281544b0 2024-03-23T15:53:30.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.928+0000 7ffb1e7fc640 1 -- 172.21.15.129:0/4191876713 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ffb20039090 con 0x7ffb281544b0 2024-03-23T15:53:30.931 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:30.931 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:30.932 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:30.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffb0c06b890 msgr2=0x7ffb0c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:30.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffb0c06b890 0x7ffb0c06dd50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb24006a90 tx=0x7ffb24006a20 comp rx=0 tx=0).stop 2024-03-23T15:53:30.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb281544b0 msgr2=0x7ffb281f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb281544b0 0x7ffb281f1ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb20033000 tx=0x7ffb20004a40 comp rx=0 tx=0).stop 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 shutdown_connections 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb281544b0 0x7ffb281f1ed0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb280caa00 0x7ffb281f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffb0c06b890 0x7ffb0c06dd50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 --2- 172.21.15.129:0/4191876713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb280c6000 0x7ffb28147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 >> 172.21.15.129:0/4191876713 conn(0x7ffb28061b50 msgr2=0x7ffb280b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 shutdown_connections 2024-03-23T15:53:30.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:30.930+0000 7ffb2fed4640 1 -- 172.21.15.129:0/4191876713 wait complete. 2024-03-23T15:53:31.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:32.003 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:33.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:33.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.103+0000 7fc9a0313640 1 Processor -- start 2024-03-23T15:53:33.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.103+0000 7fc9a0313640 1 -- start start 2024-03-23T15:53:33.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.104+0000 7fc9a0313640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc99815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:33.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.104+0000 7fc9a0313640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc99815de40 0x7fc99815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:33.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.104+0000 7fc9a0313640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc99806c830 con 0x7fc99806c030 2024-03-23T15:53:33.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.104+0000 7fc9a0313640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc9980cc370 con 0x7fc99815de40 2024-03-23T15:53:33.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.104+0000 7fc9a0313640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc99806cae0 con 0x7fc99806c410 2024-03-23T15:53:33.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.105+0000 7fc99d8ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc99815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:33.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.105+0000 7fc99e8b0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc99815de40 0x7fc99815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:33.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.106+0000 7fc99e0af640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc99806c030 0x7fc9980cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:59510/0 (socket says 172.21.15.129:59510) 2024-03-23T15:53:33.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.106+0000 7fc99e0af640 1 -- 172.21.15.129:0/1579697609 learned_addr learned my addr 172.21.15.129:0/1579697609 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:33.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.109+0000 7fc99d8ae640 1 -- 172.21.15.129:0/1579697609 >> v2:172.21.15.157:3300/0 conn(0x7fc99815de40 msgr2=0x7fc99815e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:33.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.109+0000 7fc99d8ae640 1 --2- 172.21.15.129:0/1579697609 >> v2:172.21.15.157:3300/0 conn(0x7fc99815de40 0x7fc99815e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.109+0000 7fc99d8ae640 1 -- 172.21.15.129:0/1579697609 >> v1:172.21.15.157:6789/0 conn(0x7fc99806c030 legacy=0x7fc9980cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:33.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.109+0000 7fc99d8ae640 1 -- 172.21.15.129:0/1579697609 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9980cabc0 con 0x7fc99806c410 2024-03-23T15:53:33.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.111+0000 7fc99d8ae640 1 --2- 172.21.15.129:0/1579697609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc99815d900 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc9880097a0 tx=0x7fc98802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d19cb35dc72a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:33.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.114+0000 7fc99d0ad640 1 -- 172.21.15.129:0/1579697609 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988031830 con 0x7fc99806c410 2024-03-23T15:53:33.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.117+0000 7fc99d0ad640 1 -- 172.21.15.129:0/1579697609 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc988031990 con 0x7fc99806c410 2024-03-23T15:53:33.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.117+0000 7fc99d0ad640 1 -- 172.21.15.129:0/1579697609 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988031d20 con 0x7fc99806c410 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 -- 172.21.15.129:0/1579697609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 msgr2=0x7fc99815d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 --2- 172.21.15.129:0/1579697609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc99815d900 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc9880097a0 tx=0x7fc98802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 -- 172.21.15.129:0/1579697609 shutdown_connections 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 --2- 172.21.15.129:0/1579697609 >> v2:172.21.15.157:3300/0 conn(0x7fc99815de40 0x7fc99815e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 --2- 172.21.15.129:0/1579697609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc99815d900 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 -- 172.21.15.129:0/1579697609 >> 172.21.15.129:0/1579697609 conn(0x7fc9980cfb20 msgr2=0x7fc9980cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 -- 172.21.15.129:0/1579697609 shutdown_connections 2024-03-23T15:53:33.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.118+0000 7fc9a0313640 1 -- 172.21.15.129:0/1579697609 wait complete. 2024-03-23T15:53:33.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.120+0000 7fc9a0313640 1 Processor -- start 2024-03-23T15:53:33.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.120+0000 7fc9a0313640 1 -- start start 2024-03-23T15:53:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.121+0000 7fc9a0313640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 0x7fc9981577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.121+0000 7fc9a0313640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc998158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.121+0000 7fc9a0313640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc99815de40 0x7fc9981effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.121+0000 7fc9a0313640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc998157e00 con 0x7fc99806c030 2024-03-23T15:53:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.121+0000 7fc9a0313640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc998159b80 con 0x7fc99815de40 2024-03-23T15:53:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.121+0000 7fc9a0313640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc998158850 con 0x7fc99806c410 2024-03-23T15:53:33.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.122+0000 7fc99e8b0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc99815de40 0x7fc9981effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:33.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.122+0000 7fc99e0af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 0x7fc9981577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:33.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.122+0000 7fc99d8ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc998158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:33.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.122+0000 7fc99e0af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 0x7fc9981577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39050/0 (socket says 172.21.15.129:39050) 2024-03-23T15:53:33.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.122+0000 7fc99e0af640 1 -- 172.21.15.129:0/2207623053 learned_addr learned my addr 172.21.15.129:0/2207623053 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:33.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.122+0000 7fc99e8b0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc99815de40 0x7fc9981effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:56516/0 (socket says 172.21.15.129:56516) 2024-03-23T15:53:33.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.125+0000 7fc99e0af640 1 -- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 msgr2=0x7fc998158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:33.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.125+0000 7fc99e0af640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc998158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.125+0000 7fc99e0af640 1 -- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc99815de40 msgr2=0x7fc9981effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:33.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.125+0000 7fc99e0af640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc99815de40 0x7fc9981effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.125+0000 7fc99e0af640 1 -- 172.21.15.129:0/2207623053 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc988009400 con 0x7fc99806c030 2024-03-23T15:53:33.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.128+0000 7fc99e0af640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 0x7fc9981577c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c4a0 tx=0x7fc99400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:33.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.145+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994005ce0 con 0x7fc99806c030 2024-03-23T15:53:33.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.145+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc994005e40 con 0x7fc99806c030 2024-03-23T15:53:33.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.145+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994007270 con 0x7fc99806c030 2024-03-23T15:53:33.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.145+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99815aea0 con 0x7fc99806c030 2024-03-23T15:53:33.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.145+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc998158200 con 0x7fc99806c030 2024-03-23T15:53:33.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.146+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc998075740 con 0x7fc99806c030 2024-03-23T15:53:33.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.175+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc9940073d0 con 0x7fc99806c030 2024-03-23T15:53:33.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.175+0000 7fc98effd640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc97c06b860 0x7fc97c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:33.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.176+0000 7fc99d8ae640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc97c06b860 0x7fc97c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:33.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.176+0000 7fc99d8ae640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc97c06b860 0x7fc97c06dd20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc988036000 tx=0x7fc98803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:33.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.181+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc994058310 con 0x7fc99806c030 2024-03-23T15:53:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.431+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc9940545b0 con 0x7fc99806c030 2024-03-23T15:53:33.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.649+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9980d0a30 con 0x7fc99806c030 2024-03-23T15:53:33.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.664+0000 7fc98effd640 1 -- 172.21.15.129:0/2207623053 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc994021090 con 0x7fc99806c030 2024-03-23T15:53:33.667 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:33.668 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:33.668 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:33.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc97c06b860 msgr2=0x7fc97c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc97c06b860 0x7fc97c06dd20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc988036000 tx=0x7fc98803c040 comp rx=0 tx=0).stop 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 msgr2=0x7fc9981577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 0x7fc9981577c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c4a0 tx=0x7fc99400c960 comp rx=0 tx=0).stop 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 shutdown_connections 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc97c06b860 0x7fc97c06dd20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc99815de40 0x7fc9981effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99806c410 0x7fc998158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 --2- 172.21.15.129:0/2207623053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806c030 0x7fc9981577c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 >> 172.21.15.129:0/2207623053 conn(0x7fc9980cfb20 msgr2=0x7fc99815c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 shutdown_connections 2024-03-23T15:53:33.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:33.666+0000 7fc9a0313640 1 -- 172.21.15.129:0/2207623053 wait complete. 2024-03-23T15:53:34.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:34.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:35.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:35.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.388+0000 7fa91dbf4640 1 Processor -- start 2024-03-23T15:53:35.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.388+0000 7fa91dbf4640 1 -- start start 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.389+0000 7fa91dbf4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa9181544b0 0x7fa91815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.389+0000 7fa91dbf4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9180caa00 0x7fa91806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.389+0000 7fa91dbf4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9180cbcb0 con 0x7fa91806c0b0 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.389+0000 7fa91dbf4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fa9180c5be0 con 0x7fa9180caa00 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.389+0000 7fa91dbf4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa91815b280 con 0x7fa9181544b0 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.389+0000 7fa916ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9180caa00 0x7fa91806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.390+0000 7fa916ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa9180caa00 0x7fa91806bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39064/0 (socket says 172.21.15.129:39064) 2024-03-23T15:53:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.390+0000 7fa916ffd640 1 -- 172.21.15.129:0/3550434865 learned_addr learned my addr 172.21.15.129:0/3550434865 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:35.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.390+0000 7fa9177fe640 1 --2- 172.21.15.129:0/3550434865 >> v2:172.21.15.157:3300/0 conn(0x7fa9181544b0 0x7fa91815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:35.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.392+0000 7fa916ffd640 1 -- 172.21.15.129:0/3550434865 >> v2:172.21.15.157:3300/0 conn(0x7fa9181544b0 msgr2=0x7fa91815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:35.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.392+0000 7fa916ffd640 1 --2- 172.21.15.129:0/3550434865 >> v2:172.21.15.157:3300/0 conn(0x7fa9181544b0 0x7fa91815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.392+0000 7fa916ffd640 1 -- 172.21.15.129:0/3550434865 >> v1:172.21.15.157:6789/0 conn(0x7fa91806c0b0 legacy=0x7fa9180c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:35.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.392+0000 7fa916ffd640 1 -- 172.21.15.129:0/3550434865 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91815a9e0 con 0x7fa9180caa00 2024-03-23T15:53:35.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.394+0000 7fa916ffd640 1 --2- 172.21.15.129:0/3550434865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9180caa00 0x7fa91806bb70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa91806cd70 tx=0x7fa90002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882c426802c8c149 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.397+0000 7fa9167fc640 1 -- 172.21.15.129:0/3550434865 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90002f850 con 0x7fa9180caa00 2024-03-23T15:53:35.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.397+0000 7fa9167fc640 1 -- 172.21.15.129:0/3550434865 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90002f9b0 con 0x7fa9180caa00 2024-03-23T15:53:35.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.397+0000 7fa9167fc640 1 -- 172.21.15.129:0/3550434865 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90002fd40 con 0x7fa9180caa00 2024-03-23T15:53:35.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3550434865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9180caa00 msgr2=0x7fa91806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:35.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3550434865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9180caa00 0x7fa91806bb70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa91806cd70 tx=0x7fa90002ba30 comp rx=0 tx=0).stop 2024-03-23T15:53:35.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3550434865 shutdown_connections 2024-03-23T15:53:35.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3550434865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9180caa00 0x7fa91806bb70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3550434865 >> v2:172.21.15.157:3300/0 conn(0x7fa9181544b0 0x7fa91815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3550434865 >> 172.21.15.129:0/3550434865 conn(0x7fa918061b50 msgr2=0x7fa918061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:35.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3550434865 shutdown_connections 2024-03-23T15:53:35.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.398+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3550434865 wait complete. 2024-03-23T15:53:35.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.399+0000 7fa91dbf4640 1 Processor -- start 2024-03-23T15:53:35.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.399+0000 7fa91dbf4640 1 -- start start 2024-03-23T15:53:35.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa91dbf4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 0x7fa9181f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa91dbf4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9180caa00 0x7fa918158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa91dbf4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181544b0 0x7fa9181591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa91dbf4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa9180d09c0 con 0x7fa91806c0b0 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa91dbf4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa918157a00 con 0x7fa9180caa00 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa91dbf4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa9180d04a0 con 0x7fa9181544b0 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.400+0000 7fa9177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 0x7fa9181f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.401+0000 7fa916ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9180caa00 0x7fa918158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.401+0000 7fa9177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 0x7fa9181f39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39072/0 (socket says 172.21.15.129:39072) 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.401+0000 7fa9177fe640 1 -- 172.21.15.129:0/3365477035 learned_addr learned my addr 172.21.15.129:0/3365477035 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:35.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.401+0000 7fa917fff640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181544b0 0x7fa9181591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:35.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.404+0000 7fa9177fe640 1 -- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181544b0 msgr2=0x7fa9181591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:35.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.404+0000 7fa9177fe640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181544b0 0x7fa9181591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.404+0000 7fa9177fe640 1 -- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9180caa00 msgr2=0x7fa918158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:35.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.404+0000 7fa9177fe640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9180caa00 0x7fa918158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.404+0000 7fa9177fe640 1 -- 172.21.15.129:0/3365477035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9000094d0 con 0x7fa91806c0b0 2024-03-23T15:53:35.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.406+0000 7fa9177fe640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 0x7fa9181f39e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa90800d650 tx=0x7fa90800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.410+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90800b6c0 con 0x7fa91806c0b0 2024-03-23T15:53:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.410+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90800b820 con 0x7fa91806c0b0 2024-03-23T15:53:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.410+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9180d0720 con 0x7fa91806c0b0 2024-03-23T15:53:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.410+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90800bbd0 con 0x7fa91806c0b0 2024-03-23T15:53:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.410+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa918156ea0 con 0x7fa91806c0b0 2024-03-23T15:53:35.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.411+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa918075740 con 0x7fa91806c0b0 2024-03-23T15:53:35.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.431+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa90800bd30 con 0x7fa91806c0b0 2024-03-23T15:53:35.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.431+0000 7fa90ffff640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa8f806b810 0x7fa8f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:35.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.432+0000 7fa916ffd640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa8f806b810 0x7fa8f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:35.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.432+0000 7fa916ffd640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa8f806b810 0x7fa8f806dcd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa900036000 tx=0x7fa900008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:35.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.684+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa908059960 con 0x7fa91806c0b0 2024-03-23T15:53:35.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.684+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa908055c00 con 0x7fa91806c0b0 2024-03-23T15:53:35.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.902+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9180cb540 con 0x7fa91806c0b0 2024-03-23T15:53:35.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.915+0000 7fa90ffff640 1 -- 172.21.15.129:0/3365477035 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa9080592c0 con 0x7fa91806c0b0 2024-03-23T15:53:35.918 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:35.919 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:35.919 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:35.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.917+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa8f806b810 msgr2=0x7fa8f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:35.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.917+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa8f806b810 0x7fa8f806dcd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa900036000 tx=0x7fa900008030 comp rx=0 tx=0).stop 2024-03-23T15:53:35.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.917+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 msgr2=0x7fa9181f39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.917+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 0x7fa9181f39e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa90800d650 tx=0x7fa90800db10 comp rx=0 tx=0).stop 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 shutdown_connections 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181544b0 0x7fa9181591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa8f806b810 0x7fa8f806dcd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9180caa00 0x7fa918158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 --2- 172.21.15.129:0/3365477035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa91806c0b0 0x7fa9181f39e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 >> 172.21.15.129:0/3365477035 conn(0x7fa918061b50 msgr2=0x7fa91814a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:35.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 shutdown_connections 2024-03-23T15:53:35.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:35.918+0000 7fa91dbf4640 1 -- 172.21.15.129:0/3365477035 wait complete. 2024-03-23T15:53:36.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:36.537 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:37.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:53:37.323 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:53:37.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 Processor -- start 2024-03-23T15:53:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 -- start start 2024-03-23T15:53:37.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2ce006c030 0x7f2ce006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:37.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce00cbc60 0x7f2ce015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:37.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce006cae0 con 0x7f2ce00cc080 2024-03-23T15:53:37.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f2ce00cabc0 con 0x7f2ce006c030 2024-03-23T15:53:37.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.638+0000 7f2ce4c55640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce00caea0 con 0x7f2ce00cbc60 2024-03-23T15:53:37.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.639+0000 7f2cde59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2ce006c030 0x7f2ce006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:37.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.639+0000 7f2cddd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce00cbc60 0x7f2ce015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:37.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.639+0000 7f2cded9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2ce00cc080 0x7f2ce015de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:59528/0 (socket says 172.21.15.129:59528) 2024-03-23T15:53:37.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.639+0000 7f2cded9d640 1 -- 172.21.15.129:0/2700864549 learned_addr learned my addr 172.21.15.129:0/2700864549 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:37.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.642+0000 7f2cde59c640 1 -- 172.21.15.129:0/2700864549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce00cbc60 msgr2=0x7f2ce015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:37.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.642+0000 7f2cde59c640 1 --2- 172.21.15.129:0/2700864549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce00cbc60 0x7f2ce015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.642+0000 7f2cde59c640 1 -- 172.21.15.129:0/2700864549 >> v1:172.21.15.157:6789/0 conn(0x7f2ce00cc080 legacy=0x7f2ce015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.642+0000 7f2cde59c640 1 -- 172.21.15.129:0/2700864549 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce00cb180 con 0x7f2ce006c030 2024-03-23T15:53:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.644+0000 7f2cde59c640 1 --2- 172.21.15.129:0/2700864549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce006c030 0x7f2ce006c410 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00097a0 tx=0x7f2cd002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383f1e698ecf83fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:37.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.647+0000 7f2cdd59a640 1 -- 172.21.15.129:0/2700864549 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd002e070 con 0x7f2ce006c030 2024-03-23T15:53:37.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.647+0000 7f2cdd59a640 1 -- 172.21.15.129:0/2700864549 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd00315a0 con 0x7f2ce006c030 2024-03-23T15:53:37.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.647+0000 7f2cdd59a640 1 -- 172.21.15.129:0/2700864549 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0031930 con 0x7f2ce006c030 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 -- 172.21.15.129:0/2700864549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce006c030 msgr2=0x7f2ce006c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/2700864549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce006c030 0x7f2ce006c410 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00097a0 tx=0x7f2cd002b840 comp rx=0 tx=0).stop 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 -- 172.21.15.129:0/2700864549 shutdown_connections 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/2700864549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce00cbc60 0x7f2ce015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/2700864549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce006c030 0x7f2ce006c410 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 -- 172.21.15.129:0/2700864549 >> 172.21.15.129:0/2700864549 conn(0x7f2ce00cfb20 msgr2=0x7f2ce00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 -- 172.21.15.129:0/2700864549 shutdown_connections 2024-03-23T15:53:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.648+0000 7f2ce4c55640 1 -- 172.21.15.129:0/2700864549 wait complete. 2024-03-23T15:53:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 Processor -- start 2024-03-23T15:53:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 -- start start 2024-03-23T15:53:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce006c030 0x7f2ce0151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ce00cbc60 0x7f2ce0150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 0x7f2ce00b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01512a0 con 0x7f2ce00cc080 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce01f3d10 con 0x7f2ce00cbc60 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.650+0000 7f2ce4c55640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce01f3100 con 0x7f2ce006c030 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.651+0000 7f2cded9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 0x7f2ce00b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.651+0000 7f2cde59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce006c030 0x7f2ce0151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.651+0000 7f2cded9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 0x7f2ce00b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39084/0 (socket says 172.21.15.129:39084) 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.651+0000 7f2cded9d640 1 -- 172.21.15.129:0/1471078357 learned_addr learned my addr 172.21.15.129:0/1471078357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.651+0000 7f2cddd9b640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ce00cbc60 0x7f2ce0150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:37.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.654+0000 7f2cded9d640 1 -- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce006c030 msgr2=0x7f2ce0151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:37.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.654+0000 7f2cded9d640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce006c030 0x7f2ce0151440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:37.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.654+0000 7f2cded9d640 1 -- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ce00cbc60 msgr2=0x7f2ce0150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:37.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.654+0000 7f2cded9d640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ce00cbc60 0x7f2ce0150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:37.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.654+0000 7f2cded9d640 1 -- 172.21.15.129:0/1471078357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd0009400 con 0x7f2ce00cc080 2024-03-23T15:53:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.657+0000 7f2cded9d640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 0x7f2ce00b0820 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800cbc0 tx=0x7f2cd800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:37.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.660+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd800ad40 con 0x7f2ce00cc080 2024-03-23T15:53:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.660+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd8012400 con 0x7f2ce00cc080 2024-03-23T15:53:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.660+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd800aea0 con 0x7f2ce00cc080 2024-03-23T15:53:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.660+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce01f33e0 con 0x7f2ce00cc080 2024-03-23T15:53:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.660+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce01f39a0 con 0x7f2ce00cc080 2024-03-23T15:53:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.662+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce0075740 con 0x7f2ce00cc080 2024-03-23T15:53:37.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.915+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2cd8007070 con 0x7f2ce00cc080 2024-03-23T15:53:37.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.915+0000 7f2cceffd640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cc006b840 0x7f2cc006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:37.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.916+0000 7f2cde59c640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cc006b840 0x7f2cc006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:37.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.916+0000 7f2cde59c640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cc006b840 0x7f2cc006dd00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0036000 tx=0x7f2cd003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:37.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.930+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2cd8058d00 con 0x7f2ce00cc080 2024-03-23T15:53:37.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:37.930+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2cd8054fa0 con 0x7f2ce00cc080 2024-03-23T15:53:38.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.147+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ce00d0a30 con 0x7f2ce00cc080 2024-03-23T15:53:38.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.161+0000 7f2cceffd640 1 -- 172.21.15.129:0/1471078357 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2cd8017090 con 0x7f2ce00cc080 2024-03-23T15:53:38.163 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:38.163 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:38.163 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:38.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cc006b840 msgr2=0x7f2cc006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:38.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cc006b840 0x7f2cc006dd00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0036000 tx=0x7f2cd003d040 comp rx=0 tx=0).stop 2024-03-23T15:53:38.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 msgr2=0x7f2ce00b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 0x7f2ce00b0820 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800cbc0 tx=0x7f2cd800a430 comp rx=0 tx=0).stop 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 shutdown_connections 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ce00cc080 0x7f2ce00b0820 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cc006b840 0x7f2cc006dd00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ce00cbc60 0x7f2ce0150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 --2- 172.21.15.129:0/1471078357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ce006c030 0x7f2ce0151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 >> 172.21.15.129:0/1471078357 conn(0x7f2ce00cfb20 msgr2=0x7f2ce00ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 shutdown_connections 2024-03-23T15:53:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:38.163+0000 7f2ce4c55640 1 -- 172.21.15.129:0/1471078357 wait complete. 2024-03-23T15:53:38.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:38.783 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:39.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.883+0000 7f7f1fe2a640 1 Processor -- start 2024-03-23T15:53:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 -- start start 2024-03-23T15:53:39.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180ccd60 0x7f7f1815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7f180ceb00 0x7f7f1806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7f180cef40 0x7f7f1806c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f7f1815b290 con 0x7f7f180ceb00 2024-03-23T15:53:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7f180ce800 con 0x7f7f180cef40 2024-03-23T15:53:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.884+0000 7f7f1fe2a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7f18154a70 con 0x7f7f180ccd60 2024-03-23T15:53:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.885+0000 7f7f1d3c5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7f180ceb00 0x7f7f1806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.885+0000 7f7f1dbc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180ccd60 0x7f7f1815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.885+0000 7f7f1e3c7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7f180cef40 0x7f7f1806c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.885+0000 7f7f1d3c5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7f180ceb00 0x7f7f1806c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39090/0 (socket says 172.21.15.129:39090) 2024-03-23T15:53:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.885+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/1523954129 learned_addr learned my addr 172.21.15.129:0/1523954129 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.887+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/1523954129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180ccd60 msgr2=0x7f7f1815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.887+0000 7f7f1d3c5640 1 --2- 172.21.15.129:0/1523954129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180ccd60 0x7f7f1815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.887+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/1523954129 >> v2:172.21.15.157:3300/0 conn(0x7f7f180cef40 msgr2=0x7f7f1806c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:39.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.887+0000 7f7f1d3c5640 1 --2- 172.21.15.129:0/1523954129 >> v2:172.21.15.157:3300/0 conn(0x7f7f180cef40 0x7f7f1806c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.887+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/1523954129 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f180cd950 con 0x7f7f180ceb00 2024-03-23T15:53:39.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.890+0000 7f7f1d3c5640 1 --2- 172.21.15.129:0/1523954129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f1806c270 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7f1815e8e0 tx=0x7f7f0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb200054776e230f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:39.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.892+0000 7f7f1cbc4640 1 -- 172.21.15.129:0/1523954129 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08030860 con 0x7f7f180ceb00 2024-03-23T15:53:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.894+0000 7f7f1cbc4640 1 -- 172.21.15.129:0/1523954129 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f080309c0 con 0x7f7f180ceb00 2024-03-23T15:53:39.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.894+0000 7f7f1cbc4640 1 -- 172.21.15.129:0/1523954129 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08030d50 con 0x7f7f180ceb00 2024-03-23T15:53:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/1523954129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 msgr2=0x7f7f1806c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/1523954129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f1806c270 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7f1815e8e0 tx=0x7f7f0802b9a0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/1523954129 shutdown_connections 2024-03-23T15:53:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/1523954129 >> v2:172.21.15.157:3300/0 conn(0x7f7f180cef40 0x7f7f1806c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/1523954129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f1806c270 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/1523954129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180ccd60 0x7f7f1815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/1523954129 >> 172.21.15.129:0/1523954129 conn(0x7f7f180b1840 msgr2=0x7f7f180b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/1523954129 shutdown_connections 2024-03-23T15:53:39.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.895+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/1523954129 wait complete. 2024-03-23T15:53:39.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.897+0000 7f7f1fe2a640 1 Processor -- start 2024-03-23T15:53:39.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.897+0000 7f7f1fe2a640 1 -- start start 2024-03-23T15:53:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.898+0000 7f7f1fe2a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f180ccd60 0x7f7f181f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.898+0000 7f7f1fe2a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f18154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.898+0000 7f7f1fe2a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180cef40 0x7f7f181e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.898+0000 7f7f1fe2a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f181f0640 con 0x7f7f180ceb00 2024-03-23T15:53:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.898+0000 7f7f1fe2a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18156f80 con 0x7f7f180ccd60 2024-03-23T15:53:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.898+0000 7f7f1fe2a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7f18157260 con 0x7f7f180cef40 2024-03-23T15:53:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.899+0000 7f7f1d3c5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f18154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.899+0000 7f7f1dbc6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f180ccd60 0x7f7f181f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.899+0000 7f7f1e3c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180cef40 0x7f7f181e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.899+0000 7f7f1d3c5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f18154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39100/0 (socket says 172.21.15.129:39100) 2024-03-23T15:53:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.899+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/685941503 learned_addr learned my addr 172.21.15.129:0/685941503 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:39.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.902+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180cef40 msgr2=0x7f7f181e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:39.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.902+0000 7f7f1d3c5640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180cef40 0x7f7f181e3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.902+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f180ccd60 msgr2=0x7f7f181f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:39.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.902+0000 7f7f1d3c5640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f180ccd60 0x7f7f181f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:39.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.902+0000 7f7f1d3c5640 1 -- 172.21.15.129:0/685941503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f080094d0 con 0x7f7f180ceb00 2024-03-23T15:53:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.905+0000 7f7f1d3c5640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f18154c40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7f0802b9e0 tx=0x7f7f08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:39.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.909+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08039a90 con 0x7f7f180ceb00 2024-03-23T15:53:39.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.909+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f08039c10 con 0x7f7f180ceb00 2024-03-23T15:53:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.909+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f181f08e0 con 0x7f7f180ceb00 2024-03-23T15:53:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.909+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08043410 con 0x7f7f180ceb00 2024-03-23T15:53:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.909+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f18157540 con 0x7f7f180ceb00 2024-03-23T15:53:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.910+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f1804f530 con 0x7f7f180ceb00 2024-03-23T15:53:39.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.930+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7f0804d440 con 0x7f7f180ceb00 2024-03-23T15:53:39.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.930+0000 7f7f0e7fc640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7efc06b840 0x7f7efc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:39.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.931+0000 7f7f1dbc6640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7efc06b840 0x7f7efc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:39.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.931+0000 7f7f1dbc6640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7efc06b840 0x7f7efc06dd00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009aa0 tx=0x7f7f14009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:39.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:39.933+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7f0807ad30 con 0x7f7f180ceb00 2024-03-23T15:53:40.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.179+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7f08076fd0 con 0x7f7f180ceb00 2024-03-23T15:53:40.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.398+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f1806db50 con 0x7f7f180ceb00 2024-03-23T15:53:40.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.412+0000 7f7f0e7fc640 1 -- 172.21.15.129:0/685941503 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f7f08034070 con 0x7f7f180ceb00 2024-03-23T15:53:40.415 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:40.415 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:40.415 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:40.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7efc06b840 msgr2=0x7f7efc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:40.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7efc06b840 0x7f7efc06dd00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009aa0 tx=0x7f7f14009190 comp rx=0 tx=0).stop 2024-03-23T15:53:40.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 msgr2=0x7f7f18154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:40.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f18154c40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7f0802b9e0 tx=0x7f7f08004730 comp rx=0 tx=0).stop 2024-03-23T15:53:40.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 shutdown_connections 2024-03-23T15:53:40.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f180cef40 0x7f7f181e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f180ceb00 0x7f7f18154c40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7efc06b840 0x7f7efc06dd00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 --2- 172.21.15.129:0/685941503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f180ccd60 0x7f7f181f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 >> 172.21.15.129:0/685941503 conn(0x7f7f180b1840 msgr2=0x7f7f180b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 shutdown_connections 2024-03-23T15:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:40.414+0000 7f7f1fe2a640 1 -- 172.21.15.129:0/685941503 wait complete. 2024-03-23T15:53:41.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:41.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:42.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:42.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.133+0000 7fe016977640 1 Processor -- start 2024-03-23T15:53:42.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.133+0000 7fe016977640 1 -- start start 2024-03-23T15:53:42.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.134+0000 7fe016977640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01006c410 0x7fe01015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:42.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.134+0000 7fe016977640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe01015de40 0x7fe01015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:42.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.134+0000 7fe016977640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe01006c830 con 0x7fe01006c030 2024-03-23T15:53:42.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.134+0000 7fe016977640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe0100cc370 con 0x7fe01015de40 2024-03-23T15:53:42.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.134+0000 7fe016977640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe01006cae0 con 0x7fe01006c410 2024-03-23T15:53:42.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.135+0000 7fe00ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe01006c030 0x7fe0100cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33168/0 (socket says 172.21.15.129:33168) 2024-03-23T15:53:42.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.135+0000 7fe014f14640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe01015de40 0x7fe01015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:42.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.135+0000 7fe00ffff640 1 -- 172.21.15.129:0/2765245986 learned_addr learned my addr 172.21.15.129:0/2765245986 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:42.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.135+0000 7fe00f7fe640 1 --2- 172.21.15.129:0/2765245986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01006c410 0x7fe01015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:42.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.138+0000 7fe014f14640 1 -- 172.21.15.129:0/2765245986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01006c410 msgr2=0x7fe01015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:42.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.138+0000 7fe014f14640 1 --2- 172.21.15.129:0/2765245986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01006c410 0x7fe01015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.138+0000 7fe014f14640 1 -- 172.21.15.129:0/2765245986 >> v1:172.21.15.157:6789/0 conn(0x7fe01006c030 legacy=0x7fe0100cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:42.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.138+0000 7fe014f14640 1 -- 172.21.15.129:0/2765245986 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0100cabc0 con 0x7fe01015de40 2024-03-23T15:53:42.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.140+0000 7fe014f14640 1 --2- 172.21.15.129:0/2765245986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01015de40 0x7fe01015e2d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe008009d90 tx=0x7fe00802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336ae85691cc664 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.144+0000 7fe00effd640 1 -- 172.21.15.129:0/2765245986 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00802cc50 con 0x7fe01015de40 2024-03-23T15:53:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.144+0000 7fe00effd640 1 -- 172.21.15.129:0/2765245986 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00802cdb0 con 0x7fe01015de40 2024-03-23T15:53:42.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.144+0000 7fe00effd640 1 -- 172.21.15.129:0/2765245986 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008034660 con 0x7fe01015de40 2024-03-23T15:53:42.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 -- 172.21.15.129:0/2765245986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01015de40 msgr2=0x7fe01015e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:42.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 --2- 172.21.15.129:0/2765245986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01015de40 0x7fe01015e2d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe008009d90 tx=0x7fe00802bea0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 -- 172.21.15.129:0/2765245986 shutdown_connections 2024-03-23T15:53:42.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 --2- 172.21.15.129:0/2765245986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01015de40 0x7fe01015e2d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 --2- 172.21.15.129:0/2765245986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01006c410 0x7fe01015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 -- 172.21.15.129:0/2765245986 >> 172.21.15.129:0/2765245986 conn(0x7fe0100cfb20 msgr2=0x7fe0100cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:42.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 -- 172.21.15.129:0/2765245986 shutdown_connections 2024-03-23T15:53:42.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.145+0000 7fe016977640 1 -- 172.21.15.129:0/2765245986 wait complete. 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.147+0000 7fe016977640 1 Processor -- start 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.147+0000 7fe016977640 1 -- start start 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.148+0000 7fe016977640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe01006c030 0x7fe0100b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.148+0000 7fe016977640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01006c410 0x7fe0100b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.148+0000 7fe016977640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01015de40 0x7fe0101560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.148+0000 7fe016977640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe0100c6a30 con 0x7fe01006c410 2024-03-23T15:53:42.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.148+0000 7fe016977640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe0101f3ee0 con 0x7fe01006c030 2024-03-23T15:53:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.148+0000 7fe016977640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe0101f4370 con 0x7fe01015de40 2024-03-23T15:53:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.149+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe01006c030 0x7fe0100b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.149+0000 7fe00f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01006c410 0x7fe0100b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.149+0000 7fe014f14640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01015de40 0x7fe0101560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.149+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe01006c030 0x7fe0100b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:46482/0 (socket says 172.21.15.129:46482) 2024-03-23T15:53:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.149+0000 7fe00ffff640 1 -- 172.21.15.129:0/4007145461 learned_addr learned my addr 172.21.15.129:0/4007145461 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:42.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.153+0000 7fe00f7fe640 1 -- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01015de40 msgr2=0x7fe0101560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:42.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.153+0000 7fe00f7fe640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01015de40 0x7fe0101560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.153+0000 7fe00f7fe640 1 -- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe01006c030 msgr2=0x7fe0100b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:42.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.153+0000 7fe00f7fe640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe01006c030 0x7fe0100b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.153+0000 7fe00f7fe640 1 -- 172.21.15.129:0/4007145461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe008009a70 con 0x7fe01006c410 2024-03-23T15:53:42.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.156+0000 7fe00f7fe640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01006c410 0x7fe0100b41e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe00400e850 tx=0x7fe00400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:42.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.158+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0040095b0 con 0x7fe01006c410 2024-03-23T15:53:42.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.158+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0100c6750 con 0x7fe01006c410 2024-03-23T15:53:42.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.158+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0101f3890 con 0x7fe01006c410 2024-03-23T15:53:42.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.159+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe01015bfe0 con 0x7fe01006c410 2024-03-23T15:53:42.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.160+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe004005990 con 0x7fe01006c410 2024-03-23T15:53:42.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.160+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004005d00 con 0x7fe01006c410 2024-03-23T15:53:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.424+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe00400c8e0 con 0x7fe01006c410 2024-03-23T15:53:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.424+0000 7fe00cff9640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdffc06b810 0x7fdffc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.425+0000 7fe00ffff640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdffc06b810 0x7fdffc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.425+0000 7fe00ffff640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdffc06b810 0x7fdffc06dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe000005880 tx=0x7fe000009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:42.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.438+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe0040592a0 con 0x7fe01006c410 2024-03-23T15:53:42.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.438+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe004055540 con 0x7fe01006c410 2024-03-23T15:53:42.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.661+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe010075740 con 0x7fe01006c410 2024-03-23T15:53:42.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.705+0000 7fe00cff9640 1 -- 172.21.15.129:0/4007145461 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe004021020 con 0x7fe01006c410 2024-03-23T15:53:42.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:42.708 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdffc06b810 msgr2=0x7fdffc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdffc06b810 0x7fdffc06dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe000005880 tx=0x7fe000009040 comp rx=0 tx=0).stop 2024-03-23T15:53:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01006c410 msgr2=0x7fe0100b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01006c410 0x7fe0100b41e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe00400e850 tx=0x7fe00400ed10 comp rx=0 tx=0).stop 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 shutdown_connections 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe01015de40 0x7fe0101560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe01006c410 0x7fe0100b41e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdffc06b810 0x7fdffc06dcd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 --2- 172.21.15.129:0/4007145461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe01006c030 0x7fe0100b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 >> 172.21.15.129:0/4007145461 conn(0x7fe0100cfb20 msgr2=0x7fe01015c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 shutdown_connections 2024-03-23T15:53:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:42.707+0000 7fe016977640 1 -- 172.21.15.129:0/4007145461 wait complete. 2024-03-23T15:53:43.326 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:43.326 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:44.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:44.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.426+0000 7f4f499af640 1 Processor -- start 2024-03-23T15:53:44.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.426+0000 7f4f499af640 1 -- start start 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.427+0000 7f4f499af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f441544b0 0x7f4f4415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.427+0000 7f4f499af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f440c6000 0x7f4f440c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.427+0000 7f4f499af640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f440cbcb0 con 0x7f4f440caa00 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.427+0000 7f4f499af640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f4f440c5be0 con 0x7f4f441544b0 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.427+0000 7f4f499af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f440c69b0 con 0x7f4f440c6000 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.428+0000 7f4f42ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4f441544b0 0x7f4f4415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.428+0000 7f4f427fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4f440caa00 0x7f4f4406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33178/0 (socket says 172.21.15.129:33178) 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.428+0000 7f4f427fc640 1 -- 172.21.15.129:0/1550210898 learned_addr learned my addr 172.21.15.129:0/1550210898 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.428+0000 7f4f437fe640 1 --2- 172.21.15.129:0/1550210898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f440c6000 0x7f4f440c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:44.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.430+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1550210898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f440c6000 msgr2=0x7f4f440c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:44.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.430+0000 7f4f42ffd640 1 --2- 172.21.15.129:0/1550210898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f440c6000 0x7f4f440c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:44.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.430+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1550210898 >> v1:172.21.15.157:6789/0 conn(0x7f4f440caa00 legacy=0x7f4f4406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:44.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.430+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1550210898 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4415b280 con 0x7f4f441544b0 2024-03-23T15:53:44.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.432+0000 7f4f42ffd640 1 --2- 172.21.15.129:0/1550210898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f441544b0 0x7f4f4415a2e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097a0 tx=0x7f4f3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43edf30566251ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:44.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f41ffb640 1 -- 172.21.15.129:0/1550210898 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34030610 con 0x7f4f441544b0 2024-03-23T15:53:44.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f41ffb640 1 -- 172.21.15.129:0/1550210898 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f34030770 con 0x7f4f441544b0 2024-03-23T15:53:44.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f41ffb640 1 -- 172.21.15.129:0/1550210898 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34030b00 con 0x7f4f441544b0 2024-03-23T15:53:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f499af640 1 -- 172.21.15.129:0/1550210898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f441544b0 msgr2=0x7f4f4415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f499af640 1 --2- 172.21.15.129:0/1550210898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f441544b0 0x7f4f4415a2e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097a0 tx=0x7f4f3402b960 comp rx=0 tx=0).stop 2024-03-23T15:53:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f499af640 1 -- 172.21.15.129:0/1550210898 shutdown_connections 2024-03-23T15:53:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.437+0000 7f4f499af640 1 --2- 172.21.15.129:0/1550210898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f440c6000 0x7f4f440c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.438+0000 7f4f499af640 1 --2- 172.21.15.129:0/1550210898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f441544b0 0x7f4f4415a2e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:44.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.438+0000 7f4f499af640 1 -- 172.21.15.129:0/1550210898 >> 172.21.15.129:0/1550210898 conn(0x7f4f44061b50 msgr2=0x7f4f44061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:44.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.438+0000 7f4f499af640 1 -- 172.21.15.129:0/1550210898 shutdown_connections 2024-03-23T15:53:44.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.438+0000 7f4f499af640 1 -- 172.21.15.129:0/1550210898 wait complete. 2024-03-23T15:53:44.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.439+0000 7f4f499af640 1 Processor -- start 2024-03-23T15:53:44.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.439+0000 7f4f499af640 1 -- start start 2024-03-23T15:53:44.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f499af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 0x7f4f440cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f499af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f440caa00 0x7f4f44157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f499af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f441544b0 0x7f4f440cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f499af640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441420a0 con 0x7f4f440caa00 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f499af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f440d07b0 con 0x7f4f440c6000 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f499af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f44154cf0 con 0x7f4f441544b0 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.440+0000 7f4f42ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 0x7f4f440cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.441+0000 7f4f437fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f441544b0 0x7f4f440cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.441+0000 7f4f427fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f440caa00 0x7f4f44157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.441+0000 7f4f42ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 0x7f4f440cfda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:46494/0 (socket says 172.21.15.129:46494) 2024-03-23T15:53:44.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.441+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1545032801 learned_addr learned my addr 172.21.15.129:0/1545032801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.444+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f441544b0 msgr2=0x7f4f440cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.444+0000 7f4f42ffd640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f441544b0 0x7f4f440cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.444+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1545032801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f440caa00 msgr2=0x7f4f44157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.444+0000 7f4f42ffd640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f440caa00 0x7f4f44157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.444+0000 7f4f42ffd640 1 -- 172.21.15.129:0/1545032801 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f34009400 con 0x7f4f440c6000 2024-03-23T15:53:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.447+0000 7f4f42ffd640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 0x7f4f440cfda0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4f3402bf70 tx=0x7f4f340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:44.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.455+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3403e070 con 0x7f4f440c6000 2024-03-23T15:53:44.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.455+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f34030e10 con 0x7f4f440c6000 2024-03-23T15:53:44.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.455+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34039c30 con 0x7f4f440c6000 2024-03-23T15:53:44.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.455+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f441f07a0 con 0x7f4f440c6000 2024-03-23T15:53:44.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.455+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f440c5300 con 0x7f4f440c6000 2024-03-23T15:53:44.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.456+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f44075740 con 0x7f4f440c6000 2024-03-23T15:53:44.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.480+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f340423f0 con 0x7f4f440c6000 2024-03-23T15:53:44.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.480+0000 7f4f2bfff640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f3006b840 0x7f4f3006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:44.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.481+0000 7f4f427fc640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f3006b840 0x7f4f3006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:44.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.481+0000 7f4f427fc640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f3006b840 0x7f4f3006dd00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4f4414b0e0 tx=0x7f4f38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:44.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.483+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4f3407a680 con 0x7f4f440c6000 2024-03-23T15:53:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:44.802+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f34076920 con 0x7f4f440c6000 2024-03-23T15:53:45.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.045+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f440cb540 con 0x7f4f440c6000 2024-03-23T15:53:45.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.129+0000 7f4f2bfff640 1 -- 172.21.15.129:0/1545032801 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4f34048020 con 0x7f4f440c6000 2024-03-23T15:53:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:45.132 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:45.132 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:45.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f3006b840 msgr2=0x7f4f3006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:45.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f3006b840 0x7f4f3006dd00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4f4414b0e0 tx=0x7f4f38009040 comp rx=0 tx=0).stop 2024-03-23T15:53:45.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 msgr2=0x7f4f440cfda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:45.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 0x7f4f440cfda0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4f3402bf70 tx=0x7f4f340046c0 comp rx=0 tx=0).stop 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 shutdown_connections 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f441544b0 0x7f4f440cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f440caa00 0x7f4f44157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f3006b840 0x7f4f3006dd00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 --2- 172.21.15.129:0/1545032801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f440c6000 0x7f4f440cfda0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 >> 172.21.15.129:0/1545032801 conn(0x7f4f44061b50 msgr2=0x7f4f440cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 shutdown_connections 2024-03-23T15:53:45.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:45.131+0000 7f4f499af640 1 -- 172.21.15.129:0/1545032801 wait complete. 2024-03-23T15:53:45.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:45.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:46.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.851+0000 7f1aa7ce8640 1 Processor -- start 2024-03-23T15:53:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.851+0000 7f1aa7ce8640 1 -- start start 2024-03-23T15:53:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.852+0000 7f1aa7ce8640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1aa00c6df0 0x7f1aa00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:46.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.852+0000 7f1aa7ce8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1aa00cc6a0 0x7f1aa01574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.852+0000 7f1aa7ce8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa00c3770 con 0x7f1aa013cfd0 2024-03-23T15:53:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.852+0000 7f1aa7ce8640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa0157070 con 0x7f1aa00c6df0 2024-03-23T15:53:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.853+0000 7f1aa7ce8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa00c3a50 con 0x7f1aa00cc6a0 2024-03-23T15:53:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.853+0000 7f1aa5a84640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f1aa013cfd0 0x7f1aa006c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:42808/0 (socket says 172.21.15.129:42808) 2024-03-23T15:53:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.853+0000 7f1aa5a84640 1 -- 172.21.15.129:0/2661267177 learned_addr learned my addr 172.21.15.129:0/2661267177 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.853+0000 7f1aa6285640 1 --2- 172.21.15.129:0/2661267177 >> v2:172.21.15.157:3300/0 conn(0x7f1aa00cc6a0 0x7f1aa01574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.853+0000 7f1aa5283640 1 --2- 172.21.15.129:0/2661267177 >> v2:172.21.15.129:3300/0 conn(0x7f1aa00c6df0 0x7f1aa00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:46.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.857+0000 7f1aa6285640 1 -- 172.21.15.129:0/2661267177 >> v2:172.21.15.129:3300/0 conn(0x7f1aa00c6df0 msgr2=0x7f1aa00cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:46.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.857+0000 7f1aa6285640 1 --2- 172.21.15.129:0/2661267177 >> v2:172.21.15.129:3300/0 conn(0x7f1aa00c6df0 0x7f1aa00cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:46.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.857+0000 7f1aa6285640 1 -- 172.21.15.129:0/2661267177 >> v1:172.21.15.129:6789/0 conn(0x7f1aa013cfd0 legacy=0x7f1aa006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:46.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.857+0000 7f1aa6285640 1 -- 172.21.15.129:0/2661267177 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa00c3d30 con 0x7f1aa00cc6a0 2024-03-23T15:53:46.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.860+0000 7f1aa6285640 1 --2- 172.21.15.129:0/2661267177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa00cc6a0 0x7f1aa01574a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1aa006cb80 tx=0x7f1a9802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9f25d33a5c40d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.866+0000 7f1aa4a82640 1 -- 172.21.15.129:0/2661267177 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9802cc80 con 0x7f1aa00cc6a0 2024-03-23T15:53:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.866+0000 7f1aa4a82640 1 -- 172.21.15.129:0/2661267177 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9802cde0 con 0x7f1aa00cc6a0 2024-03-23T15:53:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.866+0000 7f1aa4a82640 1 -- 172.21.15.129:0/2661267177 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98034660 con 0x7f1aa00cc6a0 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.866+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/2661267177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa00cc6a0 msgr2=0x7f1aa01574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.866+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/2661267177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa00cc6a0 0x7f1aa01574a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1aa006cb80 tx=0x7f1a9802bed0 comp rx=0 tx=0).stop 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.867+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/2661267177 shutdown_connections 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.867+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/2661267177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa00cc6a0 0x7f1aa01574a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.867+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/2661267177 >> v2:172.21.15.129:3300/0 conn(0x7f1aa00c6df0 0x7f1aa00cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.867+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/2661267177 >> 172.21.15.129:0/2661267177 conn(0x7f1aa0071dc0 msgr2=0x7f1aa00721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.867+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/2661267177 shutdown_connections 2024-03-23T15:53:46.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.867+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/2661267177 wait complete. 2024-03-23T15:53:46.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.869+0000 7f1aa7ce8640 1 Processor -- start 2024-03-23T15:53:46.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.869+0000 7f1aa7ce8640 1 -- start start 2024-03-23T15:53:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa7ce8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 0x7f1aa01ed830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa7ce8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1aa00cc6a0 0x7f1aa01581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa7ce8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa013cfd0 0x7f1aa00c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa7ce8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa01ed410 con 0x7f1aa00c6df0 2024-03-23T15:53:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa7ce8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa015a2e0 con 0x7f1aa013cfd0 2024-03-23T15:53:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa7ce8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa015a580 con 0x7f1aa00cc6a0 2024-03-23T15:53:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.870+0000 7f1aa5a84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 0x7f1aa01ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.871+0000 7f1aa5a84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 0x7f1aa01ed830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42080/0 (socket says 172.21.15.129:42080) 2024-03-23T15:53:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.871+0000 7f1aa5a84640 1 -- 172.21.15.129:0/660717705 learned_addr learned my addr 172.21.15.129:0/660717705 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.871+0000 7f1aa5283640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1aa00cc6a0 0x7f1aa01581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.872+0000 7f1aa6285640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa013cfd0 0x7f1aa00c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.874+0000 7f1aa5a84640 1 -- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1aa00cc6a0 msgr2=0x7f1aa01581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.874+0000 7f1aa5a84640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1aa00cc6a0 0x7f1aa01581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.874+0000 7f1aa5a84640 1 -- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa013cfd0 msgr2=0x7f1aa00c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.874+0000 7f1aa5a84640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa013cfd0 0x7f1aa00c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.874+0000 7f1aa5a84640 1 -- 172.21.15.129:0/660717705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a98009a70 con 0x7f1aa00c6df0 2024-03-23T15:53:46.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.876+0000 7f1aa5a84640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 0x7f1aa01ed830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00c3c0 tx=0x7f1a9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:46.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.879+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c005ce0 con 0x7f1aa00c6df0 2024-03-23T15:53:46.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.879+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa015a840 con 0x7f1aa00c6df0 2024-03-23T15:53:46.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.879+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa01ed030 con 0x7f1aa00c6df0 2024-03-23T15:53:46.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.880+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa0075740 con 0x7f1aa00c6df0 2024-03-23T15:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.882+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9c005e40 con 0x7f1aa00c6df0 2024-03-23T15:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:46.882+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c007270 con 0x7f1aa00c6df0 2024-03-23T15:53:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.148+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a9c0073d0 con 0x7f1aa00c6df0 2024-03-23T15:53:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.149+0000 7f1a967fc640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b840 0x7f1a8406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:47.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.149+0000 7f1aa5283640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b840 0x7f1a8406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:47.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.150+0000 7f1aa5283640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b840 0x7f1a8406dd00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1aa01567b0 tx=0x7f1a90009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:47.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.163+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1a9c0586c0 con 0x7f1aa00c6df0 2024-03-23T15:53:47.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.163+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a9c054960 con 0x7f1aa00c6df0 2024-03-23T15:53:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.386+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aa01ed1c0 con 0x7f1aa00c6df0 2024-03-23T15:53:47.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.406+0000 7f1a967fc640 1 -- 172.21.15.129:0/660717705 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1a9c021020 con 0x7f1aa00c6df0 2024-03-23T15:53:47.409 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:47.409 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:47.409 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:47.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b840 msgr2=0x7f1a8406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:47.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b840 0x7f1a8406dd00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1aa01567b0 tx=0x7f1a90009110 comp rx=0 tx=0).stop 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 msgr2=0x7f1aa01ed830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 0x7f1aa01ed830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00c3c0 tx=0x7f1a9c00c880 comp rx=0 tx=0).stop 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 shutdown_connections 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b840 0x7f1a8406dd00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1aa013cfd0 0x7f1aa00c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1aa00cc6a0 0x7f1aa01581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 --2- 172.21.15.129:0/660717705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa00c6df0 0x7f1aa01ed830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 >> 172.21.15.129:0/660717705 conn(0x7f1aa0071dc0 msgr2=0x7f1aa013b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 shutdown_connections 2024-03-23T15:53:47.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:47.408+0000 7f1aa7ce8640 1 -- 172.21.15.129:0/660717705 wait complete. 2024-03-23T15:53:48.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:48.027 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:49.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 Processor -- start 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 -- start start 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcde80ccd60 0x7fcde815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcde80ceb00 0x7fcde806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde815b290 con 0x7fcde80cef40 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fcde80ce800 con 0x7fcde80ccd60 2024-03-23T15:53:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.129+0000 7fcdee5f4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcde8154a70 con 0x7fcde80ceb00 2024-03-23T15:53:49.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.130+0000 7fcde7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcde80ccd60 0x7fcde815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:49.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.130+0000 7fcde7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcde80ccd60 0x7fcde815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42092/0 (socket says 172.21.15.129:42092) 2024-03-23T15:53:49.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.130+0000 7fcde7fff640 1 -- 172.21.15.129:0/2757371098 learned_addr learned my addr 172.21.15.129:0/2757371098 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.131+0000 7fcde77fe640 1 --2- 172.21.15.129:0/2757371098 >> v2:172.21.15.157:3300/0 conn(0x7fcde80ceb00 0x7fcde806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.133+0000 7fcde7fff640 1 -- 172.21.15.129:0/2757371098 >> v2:172.21.15.157:3300/0 conn(0x7fcde80ceb00 msgr2=0x7fcde806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:49.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.133+0000 7fcde7fff640 1 --2- 172.21.15.129:0/2757371098 >> v2:172.21.15.157:3300/0 conn(0x7fcde80ceb00 0x7fcde806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.133+0000 7fcde7fff640 1 -- 172.21.15.129:0/2757371098 >> v1:172.21.15.157:6789/0 conn(0x7fcde80cef40 legacy=0x7fcde815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:49.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.133+0000 7fcde7fff640 1 -- 172.21.15.129:0/2757371098 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde80cd950 con 0x7fcde80ccd60 2024-03-23T15:53:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.135+0000 7fcde7fff640 1 --2- 172.21.15.129:0/2757371098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ccd60 0x7fcde815ad50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcde808d1a0 tx=0x7fcdd802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a7d99284de4a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:49.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.137+0000 7fcde6ffd640 1 -- 172.21.15.129:0/2757371098 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd802f850 con 0x7fcde80ccd60 2024-03-23T15:53:49.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcde6ffd640 1 -- 172.21.15.129:0/2757371098 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd802f9b0 con 0x7fcde80ccd60 2024-03-23T15:53:49.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcde6ffd640 1 -- 172.21.15.129:0/2757371098 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd802fd40 con 0x7fcde80ccd60 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcdee5f4640 1 -- 172.21.15.129:0/2757371098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ccd60 msgr2=0x7fcde815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/2757371098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ccd60 0x7fcde815ad50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcde808d1a0 tx=0x7fcdd802ba30 comp rx=0 tx=0).stop 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcdee5f4640 1 -- 172.21.15.129:0/2757371098 shutdown_connections 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/2757371098 >> v2:172.21.15.157:3300/0 conn(0x7fcde80ceb00 0x7fcde806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/2757371098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ccd60 0x7fcde815ad50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.139+0000 7fcdee5f4640 1 -- 172.21.15.129:0/2757371098 >> 172.21.15.129:0/2757371098 conn(0x7fcde80b1840 msgr2=0x7fcde80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.140+0000 7fcdee5f4640 1 -- 172.21.15.129:0/2757371098 shutdown_connections 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.140+0000 7fcdee5f4640 1 -- 172.21.15.129:0/2757371098 wait complete. 2024-03-23T15:53:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.141+0000 7fcdee5f4640 1 Processor -- start 2024-03-23T15:53:49.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.141+0000 7fcdee5f4640 1 -- start start 2024-03-23T15:53:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdee5f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcde80ccd60 0x7fcde81e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdee5f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ceb00 0x7fcde8153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdee5f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcde80cef40 0x7fcde8159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdee5f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8156db0 con 0x7fcde80ceb00 2024-03-23T15:53:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdee5f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81f40e0 con 0x7fcde80cef40 2024-03-23T15:53:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdee5f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcde81f0140 con 0x7fcde80ccd60 2024-03-23T15:53:49.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcde77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ceb00 0x7fcde8153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:49.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcdecb91640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcde80cef40 0x7fcde8159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:49.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.142+0000 7fcde7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcde80ccd60 0x7fcde81e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:49.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.143+0000 7fcde77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ceb00 0x7fcde8153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42106/0 (socket says 172.21.15.129:42106) 2024-03-23T15:53:49.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.143+0000 7fcde77fe640 1 -- 172.21.15.129:0/1345023316 learned_addr learned my addr 172.21.15.129:0/1345023316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:49.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.147+0000 7fcdecb91640 1 -- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcde80ccd60 msgr2=0x7fcde81e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:49.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.147+0000 7fcdecb91640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcde80ccd60 0x7fcde81e6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.147+0000 7fcdecb91640 1 -- 172.21.15.129:0/1345023316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ceb00 msgr2=0x7fcde8153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:49.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.147+0000 7fcdecb91640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ceb00 0x7fcde8153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.147+0000 7fcdecb91640 1 -- 172.21.15.129:0/1345023316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd80094d0 con 0x7fcde80cef40 2024-03-23T15:53:49.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.151+0000 7fcdecb91640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcde80cef40 0x7fcde8159150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcde000c6c0 tx=0x7fcde000cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.155+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde0003b70 con 0x7fcde80cef40 2024-03-23T15:53:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.155+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde0003cd0 con 0x7fcde80cef40 2024-03-23T15:53:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.155+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00052f0 con 0x7fcde80cef40 2024-03-23T15:53:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.155+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde81f0420 con 0x7fcde80cef40 2024-03-23T15:53:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.155+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde81f09e0 con 0x7fcde80cef40 2024-03-23T15:53:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.156+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcde804f530 con 0x7fcde80cef40 2024-03-23T15:53:49.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.424+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcde0005450 con 0x7fcde80cef40 2024-03-23T15:53:49.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.425+0000 7fcde4ff9640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcdd406b860 0x7fcdd406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:49.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.425+0000 7fcde7fff640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcdd406b860 0x7fcdd406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:49.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.426+0000 7fcde7fff640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcdd406b860 0x7fcdd406dd20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8036000 tx=0x7fcdd8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:49.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.431+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcde0058f50 con 0x7fcde80cef40 2024-03-23T15:53:49.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.450+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcde00551f0 con 0x7fcde80cef40 2024-03-23T15:53:49.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.671+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcde806db50 con 0x7fcde80cef40 2024-03-23T15:53:49.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.693+0000 7fcde4ff9640 1 -- 172.21.15.129:0/1345023316 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcde00588b0 con 0x7fcde80cef40 2024-03-23T15:53:49.697 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:49.697 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:49.699 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:49.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcdd406b860 msgr2=0x7fcdd406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:49.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcdd406b860 0x7fcdd406dd20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8036000 tx=0x7fcdd8008030 comp rx=0 tx=0).stop 2024-03-23T15:53:49.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcde80cef40 msgr2=0x7fcde8159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcde80cef40 0x7fcde8159150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcde000c6c0 tx=0x7fcde000cb80 comp rx=0 tx=0).stop 2024-03-23T15:53:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 shutdown_connections 2024-03-23T15:53:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcdd406b860 0x7fcdd406dd20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcde80cef40 0x7fcde8159150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcde80ceb00 0x7fcde8153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 --2- 172.21.15.129:0/1345023316 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcde80ccd60 0x7fcde81e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:49.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 >> 172.21.15.129:0/1345023316 conn(0x7fcde80b1840 msgr2=0x7fcde80b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:49.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 shutdown_connections 2024-03-23T15:53:49.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:49.695+0000 7fcdee5f4640 1 -- 172.21.15.129:0/1345023316 wait complete. 2024-03-23T15:53:50.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:50.314 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:51.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.415+0000 7f80ed8fb640 1 Processor -- start 2024-03-23T15:53:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.415+0000 7f80ed8fb640 1 -- start start 2024-03-23T15:53:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80ed8fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e81544b0 0x7f80e815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80ed8fb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f80e80c6000 0x7f80e80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80ed8fb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e80cbcb0 con 0x7f80e80caa00 2024-03-23T15:53:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80ed8fb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f80e80c5be0 con 0x7f80e80c6000 2024-03-23T15:53:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80ed8fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f80e80c69b0 con 0x7f80e81544b0 2024-03-23T15:53:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e81544b0 0x7f80e815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80e67fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f80e80caa00 0x7f80e806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33980/0 (socket says 172.21.15.129:33980) 2024-03-23T15:53:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.416+0000 7f80e67fc640 1 -- 172.21.15.129:0/274869483 learned_addr learned my addr 172.21.15.129:0/274869483 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.417+0000 7f80e77fe640 1 --2- 172.21.15.129:0/274869483 >> v2:172.21.15.129:3300/0 conn(0x7f80e80c6000 0x7f80e80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.419+0000 7f80e6ffd640 1 -- 172.21.15.129:0/274869483 >> v2:172.21.15.129:3300/0 conn(0x7f80e80c6000 msgr2=0x7f80e80c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.419+0000 7f80e6ffd640 1 --2- 172.21.15.129:0/274869483 >> v2:172.21.15.129:3300/0 conn(0x7f80e80c6000 0x7f80e80c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.419+0000 7f80e6ffd640 1 -- 172.21.15.129:0/274869483 >> v1:172.21.15.157:6789/0 conn(0x7f80e80caa00 legacy=0x7f80e806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.419+0000 7f80e6ffd640 1 -- 172.21.15.129:0/274869483 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e815b280 con 0x7f80e81544b0 2024-03-23T15:53:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.422+0000 7f80e6ffd640 1 --2- 172.21.15.129:0/274869483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e81544b0 0x7f80e815a2e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80d80097a0 tx=0x7f80d802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20fbb24dd4ffd808 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80e5ffb640 1 -- 172.21.15.129:0/274869483 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8030610 con 0x7f80e81544b0 2024-03-23T15:53:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80e5ffb640 1 -- 172.21.15.129:0/274869483 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d8030770 con 0x7f80e81544b0 2024-03-23T15:53:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80e5ffb640 1 -- 172.21.15.129:0/274869483 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8030b00 con 0x7f80e81544b0 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 -- 172.21.15.129:0/274869483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e81544b0 msgr2=0x7f80e815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/274869483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e81544b0 0x7f80e815a2e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80d80097a0 tx=0x7f80d802b960 comp rx=0 tx=0).stop 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 -- 172.21.15.129:0/274869483 shutdown_connections 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/274869483 >> v2:172.21.15.129:3300/0 conn(0x7f80e80c6000 0x7f80e80c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/274869483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e81544b0 0x7f80e815a2e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 -- 172.21.15.129:0/274869483 >> 172.21.15.129:0/274869483 conn(0x7f80e8061b50 msgr2=0x7f80e8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 -- 172.21.15.129:0/274869483 shutdown_connections 2024-03-23T15:53:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.426+0000 7f80ed8fb640 1 -- 172.21.15.129:0/274869483 wait complete. 2024-03-23T15:53:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 Processor -- start 2024-03-23T15:53:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 -- start start 2024-03-23T15:53:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 0x7f80e80ce210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e80caa00 0x7f80e81f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80e81544b0 0x7f80e80d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80e8145950 con 0x7f80e80c6000 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.428+0000 7f80ed8fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f80e81489c0 con 0x7f80e81544b0 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.429+0000 7f80ed8fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f80e8157880 con 0x7f80e80caa00 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.429+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 0x7f80e80ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.429+0000 7f80e67fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e80caa00 0x7f80e81f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.429+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80e81544b0 0x7f80e80d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.429+0000 7f80e6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 0x7f80e80ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46648/0 (socket says 172.21.15.129:46648) 2024-03-23T15:53:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.429+0000 7f80e6ffd640 1 -- 172.21.15.129:0/3094369374 learned_addr learned my addr 172.21.15.129:0/3094369374 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.433+0000 7f80e6ffd640 1 -- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e80caa00 msgr2=0x7f80e81f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.433+0000 7f80e6ffd640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e80caa00 0x7f80e81f05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.433+0000 7f80e6ffd640 1 -- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80e81544b0 msgr2=0x7f80e80d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.433+0000 7f80e6ffd640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80e81544b0 0x7f80e80d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.433+0000 7f80e6ffd640 1 -- 172.21.15.129:0/3094369374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d8009400 con 0x7f80e80c6000 2024-03-23T15:53:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.435+0000 7f80e6ffd640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 0x7f80e80ce210 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80d80098c0 tx=0x7f80d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.439+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d803e070 con 0x7f80e80c6000 2024-03-23T15:53:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.439+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d8030e10 con 0x7f80e80c6000 2024-03-23T15:53:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.439+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8039c50 con 0x7f80e80c6000 2024-03-23T15:53:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.439+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e8148400 con 0x7f80e80c6000 2024-03-23T15:53:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.439+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e80cdb70 con 0x7f80e80c6000 2024-03-23T15:53:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.440+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e8075740 con 0x7f80e80c6000 2024-03-23T15:53:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.463+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f80d8055430 con 0x7f80e80c6000 2024-03-23T15:53:51.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.464+0000 7f80cffff640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80d406b810 0x7f80d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:51.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.464+0000 7f80e67fc640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80d406b810 0x7f80d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:51.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.464+0000 7f80e67fc640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80d406b810 0x7f80d406dcd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80e80ce750 tx=0x7f80dc007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:51.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.467+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f80d807b730 con 0x7f80e80c6000 2024-03-23T15:53:51.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.713+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f80d8077a10 con 0x7f80e80c6000 2024-03-23T15:53:51.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.936+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80e8157d00 con 0x7f80e80c6000 2024-03-23T15:53:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.950+0000 7f80cffff640 1 -- 172.21.15.129:0/3094369374 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f80d8034070 con 0x7f80e80c6000 2024-03-23T15:53:51.953 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:51.953 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80d406b810 msgr2=0x7f80d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80d406b810 0x7f80d406dcd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80e80ce750 tx=0x7f80dc007e80 comp rx=0 tx=0).stop 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 msgr2=0x7f80e80ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 0x7f80e80ce210 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80d80098c0 tx=0x7f80d80046c0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 shutdown_connections 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f80d406b810 0x7f80d406dcd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80e81544b0 0x7f80e80d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80e80caa00 0x7f80e81f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 --2- 172.21.15.129:0/3094369374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80e80c6000 0x7f80e80ce210 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:51.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 >> 172.21.15.129:0/3094369374 conn(0x7f80e8061b50 msgr2=0x7f80e80b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:51.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 shutdown_connections 2024-03-23T15:53:51.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:51.953+0000 7f80ed8fb640 1 -- 172.21.15.129:0/3094369374 wait complete. 2024-03-23T15:53:52.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:52.572 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:53.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:53.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 Processor -- start 2024-03-23T15:53:53.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 -- start start 2024-03-23T15:53:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb71c0ceb00 0x7fb71c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb71c15b290 con 0x7fb71c0cef40 2024-03-23T15:53:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c0ce800 con 0x7fb71c0ceb00 2024-03-23T15:53:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.672+0000 7fb721aeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c154a70 con 0x7fb71c0ccd60 2024-03-23T15:53:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.673+0000 7fb71b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:53.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.673+0000 7fb71bfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb71c0cef40 0x7fb71c15b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33988/0 (socket says 172.21.15.129:33988) 2024-03-23T15:53:53.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.673+0000 7fb71bfff640 1 -- 172.21.15.129:0/4072587403 learned_addr learned my addr 172.21.15.129:0/4072587403 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:53.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.674+0000 7fb71affd640 1 --2- 172.21.15.129:0/4072587403 >> v2:172.21.15.157:3300/0 conn(0x7fb71c0ceb00 0x7fb71c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.677+0000 7fb71b7fe640 1 -- 172.21.15.129:0/4072587403 >> v2:172.21.15.157:3300/0 conn(0x7fb71c0ceb00 msgr2=0x7fb71c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:53:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.677+0000 7fb71b7fe640 1 --2- 172.21.15.129:0/4072587403 >> v2:172.21.15.157:3300/0 conn(0x7fb71c0ceb00 0x7fb71c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.677+0000 7fb71b7fe640 1 -- 172.21.15.129:0/4072587403 >> v1:172.21.15.157:6789/0 conn(0x7fb71c0cef40 legacy=0x7fb71c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.677+0000 7fb71b7fe640 1 -- 172.21.15.129:0/4072587403 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c0cd950 con 0x7fb71c0ccd60 2024-03-23T15:53:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.680+0000 7fb71b7fe640 1 --2- 172.21.15.129:0/4072587403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c15ad50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0097a0 tx=0x7fb70c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b88261660d9465e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.682+0000 7fb71a7fc640 1 -- 172.21.15.129:0/4072587403 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c030610 con 0x7fb71c0ccd60 2024-03-23T15:53:53.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.685+0000 7fb71a7fc640 1 -- 172.21.15.129:0/4072587403 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70c030770 con 0x7fb71c0ccd60 2024-03-23T15:53:53.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.685+0000 7fb71a7fc640 1 -- 172.21.15.129:0/4072587403 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c030b00 con 0x7fb71c0ccd60 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.685+0000 7fb721aeb640 1 -- 172.21.15.129:0/4072587403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 msgr2=0x7fb71c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.685+0000 7fb721aeb640 1 --2- 172.21.15.129:0/4072587403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c15ad50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0097a0 tx=0x7fb70c02b960 comp rx=0 tx=0).stop 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.686+0000 7fb721aeb640 1 -- 172.21.15.129:0/4072587403 shutdown_connections 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.686+0000 7fb721aeb640 1 --2- 172.21.15.129:0/4072587403 >> v2:172.21.15.157:3300/0 conn(0x7fb71c0ceb00 0x7fb71c06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.686+0000 7fb721aeb640 1 --2- 172.21.15.129:0/4072587403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c15ad50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.686+0000 7fb721aeb640 1 -- 172.21.15.129:0/4072587403 >> 172.21.15.129:0/4072587403 conn(0x7fb71c0b1840 msgr2=0x7fb71c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.686+0000 7fb721aeb640 1 -- 172.21.15.129:0/4072587403 shutdown_connections 2024-03-23T15:53:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.686+0000 7fb721aeb640 1 -- 172.21.15.129:0/4072587403 wait complete. 2024-03-23T15:53:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.687+0000 7fb721aeb640 1 Processor -- start 2024-03-23T15:53:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.688+0000 7fb721aeb640 1 -- start start 2024-03-23T15:53:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb721aeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c1f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb721aeb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb71c0ceb00 0x7fb71c0c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb721aeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 0x7fb71c1f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb721aeb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0c7f90 con 0x7fb71c0cef40 2024-03-23T15:53:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb721aeb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c1577c0 con 0x7fb71c0ceb00 2024-03-23T15:53:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb721aeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c1f1450 con 0x7fb71c0ccd60 2024-03-23T15:53:53.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb71b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c1f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:53.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb71bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 0x7fb71c1f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:53.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.689+0000 7fb71affd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb71c0ceb00 0x7fb71c0c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:53.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.690+0000 7fb71bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 0x7fb71c1f0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46662/0 (socket says 172.21.15.129:46662) 2024-03-23T15:53:53.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.690+0000 7fb71bfff640 1 -- 172.21.15.129:0/2614795376 learned_addr learned my addr 172.21.15.129:0/2614795376 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.693+0000 7fb71bfff640 1 -- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 msgr2=0x7fb71c1f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.693+0000 7fb71bfff640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c1f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.693+0000 7fb71bfff640 1 -- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb71c0ceb00 msgr2=0x7fb71c0c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.693+0000 7fb71bfff640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb71c0ceb00 0x7fb71c0c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.693+0000 7fb71bfff640 1 -- 172.21.15.129:0/2614795376 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c009400 con 0x7fb71c0cef40 2024-03-23T15:53:53.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.696+0000 7fb71bfff640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 0x7fb71c1f0f10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb71400cbc0 tx=0x7fb71400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.698+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71400ace0 con 0x7fb71c0cef40 2024-03-23T15:53:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.698+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71c0c6dd0 con 0x7fb71c0cef40 2024-03-23T15:53:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.698+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb71c0c7310 con 0x7fb71c0cef40 2024-03-23T15:53:53.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.699+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb714001c00 con 0x7fb71c0cef40 2024-03-23T15:53:53.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.699+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71401c640 con 0x7fb71c0cef40 2024-03-23T15:53:53.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.700+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc005290 con 0x7fb71c0cef40 2024-03-23T15:53:53.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.953+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb71401c7a0 con 0x7fb71c0cef40 2024-03-23T15:53:53.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.954+0000 7fb703fff640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb70806b810 0x7fb70806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:53.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.954+0000 7fb71b7fe640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb70806b810 0x7fb70806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:53.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.954+0000 7fb71b7fe640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb70806b810 0x7fb70806dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb70c036000 tx=0x7fb70c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:53.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.968+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb71400ae40 con 0x7fb71c0cef40 2024-03-23T15:53:53.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:53.968+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb714058910 con 0x7fb71c0cef40 2024-03-23T15:53:54.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.202+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6fc0058f0 con 0x7fb71c0cef40 2024-03-23T15:53:54.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.215+0000 7fb703fff640 1 -- 172.21.15.129:0/2614795376 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb6fc0058f0 con 0x7fb71c0cef40 2024-03-23T15:53:54.219 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:54.219 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:54.219 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb70806b810 msgr2=0x7fb70806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb70806b810 0x7fb70806dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb70c036000 tx=0x7fb70c03b040 comp rx=0 tx=0).stop 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 msgr2=0x7fb71c1f0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 0x7fb71c1f0f10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb71400cbc0 tx=0x7fb71400a430 comp rx=0 tx=0).stop 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 shutdown_connections 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb71c0cef40 0x7fb71c1f0f10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb70806b810 0x7fb70806dcd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb71c0ceb00 0x7fb71c0c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 --2- 172.21.15.129:0/2614795376 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb71c0ccd60 0x7fb71c1f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 >> 172.21.15.129:0/2614795376 conn(0x7fb71c0b1840 msgr2=0x7fb71c0b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.218+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 shutdown_connections 2024-03-23T15:53:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:54.219+0000 7fb721aeb640 1 -- 172.21.15.129:0/2614795376 wait complete. 2024-03-23T15:53:54.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:54.837 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:55.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:55.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.937+0000 7fd03150c640 1 Processor -- start 2024-03-23T15:53:55.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.937+0000 7fd03150c640 1 -- start start 2024-03-23T15:53:55.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.938+0000 7fd03150c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd02c154270 0x7fd02c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:55.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.938+0000 7fd03150c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd02c15d880 con 0x7fd02c15ad10 2024-03-23T15:53:55.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.938+0000 7fd03150c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd02c0c4ae0 con 0x7fd02c0c4f70 2024-03-23T15:53:55.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.938+0000 7fd03150c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd02c0cd5e0 con 0x7fd02c154270 2024-03-23T15:53:55.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.939+0000 7fd02b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd02c0c4f70 0x7fd02c0c53e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33992/0 (socket says 172.21.15.129:33992) 2024-03-23T15:53:55.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.939+0000 7fd02b7fe640 1 -- 172.21.15.129:0/2022245322 learned_addr learned my addr 172.21.15.129:0/2022245322 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:55.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.939+0000 7fd02affd640 1 --2- 172.21.15.129:0/2022245322 >> v2:172.21.15.129:3300/0 conn(0x7fd02c154270 0x7fd02c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.942+0000 7fd029ffb640 1 -- 172.21.15.129:0/2022245322 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736448950 0 0) 0x7fd02c0c4ae0 con 0x7fd02c0c4f70 2024-03-23T15:53:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.942+0000 7fd029ffb640 1 -- 172.21.15.129:0/2022245322 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd014003520 con 0x7fd02c0c4f70 2024-03-23T15:53:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.942+0000 7fd02affd640 1 -- 172.21.15.129:0/2022245322 >> v1:172.21.15.157:6789/0 conn(0x7fd02c0c4f70 legacy=0x7fd02c0c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.942+0000 7fd02affd640 1 -- 172.21.15.129:0/2022245322 >> v1:172.21.15.129:6789/0 conn(0x7fd02c15ad10 legacy=0x7fd02c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:55.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.942+0000 7fd02affd640 1 -- 172.21.15.129:0/2022245322 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02c15dab0 con 0x7fd02c154270 2024-03-23T15:53:55.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.946+0000 7fd02affd640 1 --2- 172.21.15.129:0/2022245322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c154270 0x7fd02c0ca580 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0097a0 tx=0x7fd01c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a416ac830b396d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:55.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.949+0000 7fd029ffb640 1 -- 172.21.15.129:0/2022245322 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c02e070 con 0x7fd02c154270 2024-03-23T15:53:55.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.951+0000 7fd029ffb640 1 -- 172.21.15.129:0/2022245322 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd01c0315a0 con 0x7fd02c154270 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.951+0000 7fd03150c640 1 -- 172.21.15.129:0/2022245322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c154270 msgr2=0x7fd02c0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.951+0000 7fd03150c640 1 --2- 172.21.15.129:0/2022245322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c154270 0x7fd02c0ca580 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd01c0097a0 tx=0x7fd01c02b840 comp rx=0 tx=0).stop 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.951+0000 7fd03150c640 1 -- 172.21.15.129:0/2022245322 shutdown_connections 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.951+0000 7fd03150c640 1 --2- 172.21.15.129:0/2022245322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c154270 0x7fd02c0ca580 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.951+0000 7fd03150c640 1 -- 172.21.15.129:0/2022245322 >> 172.21.15.129:0/2022245322 conn(0x7fd02c06d290 msgr2=0x7fd02c072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.952+0000 7fd03150c640 1 -- 172.21.15.129:0/2022245322 shutdown_connections 2024-03-23T15:53:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.952+0000 7fd03150c640 1 -- 172.21.15.129:0/2022245322 wait complete. 2024-03-23T15:53:55.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.953+0000 7fd03150c640 1 Processor -- start 2024-03-23T15:53:55.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.953+0000 7fd03150c640 1 -- start start 2024-03-23T15:53:55.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.954+0000 7fd03150c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02c0c4f70 0x7fd02c149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:55.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.954+0000 7fd03150c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd02c154270 0x7fd02c146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:55.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.954+0000 7fd03150c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 0x7fd02c146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:55.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.954+0000 7fd03150c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c14f980 con 0x7fd02c15ad10 2024-03-23T15:53:55.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.954+0000 7fd03150c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c0c75f0 con 0x7fd02c154270 2024-03-23T15:53:55.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.954+0000 7fd03150c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd02c1505d0 con 0x7fd02c0c4f70 2024-03-23T15:53:55.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.955+0000 7fd02b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 0x7fd02c146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:55.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.955+0000 7fd02a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd02c154270 0x7fd02c146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:55.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.955+0000 7fd02affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02c0c4f70 0x7fd02c149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.955+0000 7fd02b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 0x7fd02c146a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46682/0 (socket says 172.21.15.129:46682) 2024-03-23T15:53:55.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.955+0000 7fd02b7fe640 1 -- 172.21.15.129:0/3764815621 learned_addr learned my addr 172.21.15.129:0/3764815621 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.960+0000 7fd02b7fe640 1 -- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02c0c4f70 msgr2=0x7fd02c149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.960+0000 7fd02b7fe640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02c0c4f70 0x7fd02c149680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.960+0000 7fd02b7fe640 1 -- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd02c154270 msgr2=0x7fd02c146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.960+0000 7fd02b7fe640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd02c154270 0x7fd02c146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.960+0000 7fd02b7fe640 1 -- 172.21.15.129:0/3764815621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c009400 con 0x7fd02c15ad10 2024-03-23T15:53:55.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.963+0000 7fd02b7fe640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 0x7fd02c146a40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd02400cad0 tx=0x7fd02400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.968+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02400ab70 con 0x7fd02c15ad10 2024-03-23T15:53:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.968+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd02400acd0 con 0x7fd02c15ad10 2024-03-23T15:53:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.968+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02c153a60 con 0x7fd02c15ad10 2024-03-23T15:53:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.968+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024012620 con 0x7fd02c15ad10 2024-03-23T15:53:55.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.968+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd02c1502f0 con 0x7fd02c15ad10 2024-03-23T15:53:55.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:55.969+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd02c0b28f0 con 0x7fd02c15ad10 2024-03-23T15:53:56.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.237+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd024007070 con 0x7fd02c15ad10 2024-03-23T15:53:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.238+0000 7fd0137fe640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd01806b890 0x7fd01806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.238+0000 7fd02affd640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd01806b890 0x7fd01806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.239+0000 7fd02affd640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd01806b890 0x7fd01806dd50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd01c036000 tx=0x7fd01c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.252+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd02400ae40 con 0x7fd02c15ad10 2024-03-23T15:53:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.252+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd024054fe0 con 0x7fd02c15ad10 2024-03-23T15:53:56.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.475+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd02c06da00 con 0x7fd02c15ad10 2024-03-23T15:53:56.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.487+0000 7fd0137fe640 1 -- 172.21.15.129:0/3764815621 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd0240586a0 con 0x7fd02c15ad10 2024-03-23T15:53:56.490 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:56.490 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.489+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd01806b890 msgr2=0x7fd01806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.489+0000 7fd03150c640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd01806b890 0x7fd01806dd50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd01c036000 tx=0x7fd01c008030 comp rx=0 tx=0).stop 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.489+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 msgr2=0x7fd02c146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.489+0000 7fd03150c640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 0x7fd02c146a40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd02400cad0 tx=0x7fd02400cf90 comp rx=0 tx=0).stop 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 shutdown_connections 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c15ad10 0x7fd02c146a40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd01806b890 0x7fd01806dd50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd02c154270 0x7fd02c146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:56.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 --2- 172.21.15.129:0/3764815621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02c0c4f70 0x7fd02c149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 >> 172.21.15.129:0/3764815621 conn(0x7fd02c06d290 msgr2=0x7fd02c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 shutdown_connections 2024-03-23T15:53:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:56.490+0000 7fd03150c640 1 -- 172.21.15.129:0/3764815621 wait complete. 2024-03-23T15:53:57.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:57.111 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:53:58.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:53:58.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.215+0000 7f3e1d981640 1 Processor -- start 2024-03-23T15:53:58.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.215+0000 7f3e1d981640 1 -- start start 2024-03-23T15:53:58.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.216+0000 7f3e1d981640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e181544b0 0x7f3e1815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:58.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.216+0000 7f3e1d981640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e180cbcb0 con 0x7f3e180c6000 2024-03-23T15:53:58.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.216+0000 7f3e1d981640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1815b280 con 0x7f3e180caa00 2024-03-23T15:53:58.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.216+0000 7f3e1d981640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e180c5be0 con 0x7f3e181544b0 2024-03-23T15:53:58.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.217+0000 7f3e177fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3e180c6000 0x7f3e18164c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43976/0 (socket says 172.21.15.129:43976) 2024-03-23T15:53:58.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.217+0000 7f3e177fe640 1 -- 172.21.15.129:0/2271722717 learned_addr learned my addr 172.21.15.129:0/2271722717 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:53:58.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.217+0000 7f3e16ffd640 1 --2- 172.21.15.129:0/2271722717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e181544b0 0x7f3e1815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:58.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.219+0000 7f3e167fc640 1 -- 172.21.15.129:0/2271722717 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2908204018 0 0) 0x7f3e180cbcb0 con 0x7f3e180c6000 2024-03-23T15:53:58.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.219+0000 7f3e167fc640 1 -- 172.21.15.129:0/2271722717 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e04003520 con 0x7f3e180c6000 2024-03-23T15:53:58.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.220+0000 7f3e16ffd640 1 -- 172.21.15.129:0/2271722717 >> v1:172.21.15.157:6789/0 conn(0x7f3e180caa00 legacy=0x7f3e1806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.220+0000 7f3e16ffd640 1 -- 172.21.15.129:0/2271722717 >> v1:172.21.15.129:6789/0 conn(0x7f3e180c6000 legacy=0x7f3e18164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.220+0000 7f3e16ffd640 1 -- 172.21.15.129:0/2271722717 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1815a9e0 con 0x7f3e181544b0 2024-03-23T15:53:58.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.222+0000 7f3e16ffd640 1 --2- 172.21.15.129:0/2271722717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e181544b0 0x7f3e1815a2e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e080097d0 tx=0x7f3e0802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d2d68cb8bc1c316 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e167fc640 1 -- 172.21.15.129:0/2271722717 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08030880 con 0x7f3e181544b0 2024-03-23T15:53:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e167fc640 1 -- 172.21.15.129:0/2271722717 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e080309e0 con 0x7f3e181544b0 2024-03-23T15:53:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e167fc640 1 -- 172.21.15.129:0/2271722717 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08030d70 con 0x7f3e181544b0 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 -- 172.21.15.129:0/2271722717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e181544b0 msgr2=0x7f3e1815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2271722717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e181544b0 0x7f3e1815a2e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e080097d0 tx=0x7f3e0802b9c0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 -- 172.21.15.129:0/2271722717 shutdown_connections 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2271722717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e181544b0 0x7f3e1815a2e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 -- 172.21.15.129:0/2271722717 >> 172.21.15.129:0/2271722717 conn(0x7f3e18061b50 msgr2=0x7f3e18061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 -- 172.21.15.129:0/2271722717 shutdown_connections 2024-03-23T15:53:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.227+0000 7f3e1d981640 1 -- 172.21.15.129:0/2271722717 wait complete. 2024-03-23T15:53:58.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.228+0000 7f3e1d981640 1 Processor -- start 2024-03-23T15:53:58.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 -- start start 2024-03-23T15:53:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e180c6000 0x7f3e18154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e180caa00 0x7f3e181e6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 0x7f3e181efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181f3380 con 0x7f3e180c6000 2024-03-23T15:53:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181f3ec0 con 0x7f3e181544b0 2024-03-23T15:53:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.229+0000 7f3e1d981640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e18155260 con 0x7f3e180caa00 2024-03-23T15:53:58.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.230+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 0x7f3e181efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:58.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.230+0000 7f3e16ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e180c6000 0x7f3e18154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:58.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.230+0000 7f3e0ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e180caa00 0x7f3e181e6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:58.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.230+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 0x7f3e181efe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:60850/0 (socket says 172.21.15.129:60850) 2024-03-23T15:53:58.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.230+0000 7f3e177fe640 1 -- 172.21.15.129:0/2110824716 learned_addr learned my addr 172.21.15.129:0/2110824716 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:53:58.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.235+0000 7f3e177fe640 1 -- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e180caa00 msgr2=0x7f3e181e6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.235+0000 7f3e177fe640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e180caa00 0x7f3e181e6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.235+0000 7f3e177fe640 1 -- 172.21.15.129:0/2110824716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e180c6000 msgr2=0x7f3e18154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.235+0000 7f3e177fe640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e180c6000 0x7f3e18154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.235+0000 7f3e177fe640 1 -- 172.21.15.129:0/2110824716 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e08009400 con 0x7f3e181544b0 2024-03-23T15:53:58.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.238+0000 7f3e177fe640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 0x7f3e181efe10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000cb40 tx=0x7f3e1000a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:58.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.243+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1000aca0 con 0x7f3e181544b0 2024-03-23T15:53:58.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.243+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e1000ae00 con 0x7f3e181544b0 2024-03-23T15:53:58.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.243+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e181f3590 con 0x7f3e181544b0 2024-03-23T15:53:58.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.243+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10012620 con 0x7f3e181544b0 2024-03-23T15:53:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.243+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e181f3b50 con 0x7f3e181544b0 2024-03-23T15:53:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.244+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e180ca770 con 0x7f3e181544b0 2024-03-23T15:53:58.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.278+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e10007070 con 0x7f3e181544b0 2024-03-23T15:53:58.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.279+0000 7f3e14ff9640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3df806b860 0x7f3df806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:53:58.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.279+0000 7f3e16ffd640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3df806b860 0x7f3df806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:53:58.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.279+0000 7f3e16ffd640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3df806b860 0x7f3df806dd20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3e08036000 tx=0x7f3e0803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:53:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.284+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3e10058e40 con 0x7f3e181544b0 2024-03-23T15:53:58.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.544+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e100550e0 con 0x7f3e181544b0 2024-03-23T15:53:58.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.781+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e18075740 con 0x7f3e181544b0 2024-03-23T15:53:58.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.796+0000 7f3e14ff9640 1 -- 172.21.15.129:0/2110824716 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3e100587a0 con 0x7f3e181544b0 2024-03-23T15:53:58.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:53:58.801 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:53:58.801 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3df806b860 msgr2=0x7f3df806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3df806b860 0x7f3df806dd20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3e08036000 tx=0x7f3e0803b040 comp rx=0 tx=0).stop 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 msgr2=0x7f3e181efe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 0x7f3e181efe10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000cb40 tx=0x7f3e1000a060 comp rx=0 tx=0).stop 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 shutdown_connections 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3df806b860 0x7f3df806dd20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e181544b0 0x7f3e181efe10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e180caa00 0x7f3e181e6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 --2- 172.21.15.129:0/2110824716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e180c6000 0x7f3e18154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:53:58.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 >> 172.21.15.129:0/2110824716 conn(0x7f3e18061b50 msgr2=0x7f3e18158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:53:58.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 shutdown_connections 2024-03-23T15:53:58.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:53:58.799+0000 7f3e1d981640 1 -- 172.21.15.129:0/2110824716 wait complete. 2024-03-23T15:53:59.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:53:59.420 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:00.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:00.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.521+0000 7fc6e291e640 1 Processor -- start 2024-03-23T15:54:00.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.521+0000 7fc6e291e640 1 -- start start 2024-03-23T15:54:00.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.522+0000 7fc6e291e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc15a2e0 0x7fc6dc06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:00.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.522+0000 7fc6e291e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc6dc06c0b0 0x7fc6dc0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:00.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.522+0000 7fc6e291e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6dc0cbcb0 con 0x7fc6dc1544b0 2024-03-23T15:54:00.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.522+0000 7fc6e291e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc6dc0c5be0 con 0x7fc6dc06c0b0 2024-03-23T15:54:00.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.522+0000 7fc6e291e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc6dc15b280 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.523+0000 7fc6e0ebb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc6dc06c0b0 0x7fc6dc0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.523+0000 7fc6db7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc15a2e0 0x7fc6dc06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.523+0000 7fc6dbfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc6dc1544b0 0x7fc6dc0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:34928/0 (socket says 172.21.15.129:34928) 2024-03-23T15:54:00.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.523+0000 7fc6dbfff640 1 -- 172.21.15.129:0/3501016589 learned_addr learned my addr 172.21.15.129:0/3501016589 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:00.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.526+0000 7fc6db7fe640 1 -- 172.21.15.129:0/3501016589 >> v2:172.21.15.129:3300/0 conn(0x7fc6dc06c0b0 msgr2=0x7fc6dc0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:00.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.526+0000 7fc6db7fe640 1 --2- 172.21.15.129:0/3501016589 >> v2:172.21.15.129:3300/0 conn(0x7fc6dc06c0b0 0x7fc6dc0c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:00.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.526+0000 7fc6db7fe640 1 -- 172.21.15.129:0/3501016589 >> v1:172.21.15.129:6789/0 conn(0x7fc6dc1544b0 legacy=0x7fc6dc0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:00.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.526+0000 7fc6db7fe640 1 -- 172.21.15.129:0/3501016589 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6dc15a9e0 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.528+0000 7fc6db7fe640 1 --2- 172.21.15.129:0/3501016589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc15a2e0 0x7fc6dc06bb70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00097a0 tx=0x7fc6d002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b781f1a30a390b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:00.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.532+0000 7fc6daffd640 1 -- 172.21.15.129:0/3501016589 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0030610 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.532+0000 7fc6daffd640 1 -- 172.21.15.129:0/3501016589 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d0030770 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.532+0000 7fc6daffd640 1 -- 172.21.15.129:0/3501016589 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0030b00 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 -- 172.21.15.129:0/3501016589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc15a2e0 msgr2=0x7fc6dc06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 --2- 172.21.15.129:0/3501016589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc15a2e0 0x7fc6dc06bb70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00097a0 tx=0x7fc6d002b960 comp rx=0 tx=0).stop 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 -- 172.21.15.129:0/3501016589 shutdown_connections 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 --2- 172.21.15.129:0/3501016589 >> v2:172.21.15.129:3300/0 conn(0x7fc6dc06c0b0 0x7fc6dc0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 --2- 172.21.15.129:0/3501016589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc15a2e0 0x7fc6dc06bb70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 -- 172.21.15.129:0/3501016589 >> 172.21.15.129:0/3501016589 conn(0x7fc6dc061b50 msgr2=0x7fc6dc061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 -- 172.21.15.129:0/3501016589 shutdown_connections 2024-03-23T15:54:00.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.533+0000 7fc6e291e640 1 -- 172.21.15.129:0/3501016589 wait complete. 2024-03-23T15:54:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.534+0000 7fc6e291e640 1 Processor -- start 2024-03-23T15:54:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.534+0000 7fc6e291e640 1 -- start start 2024-03-23T15:54:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e291e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6dc06c0b0 0x7fc6dc0d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e291e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc1544b0 0x7fc6dc146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e291e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 0x7fc6dc147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e291e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc1551a0 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e291e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc6dc1466b0 con 0x7fc6dc06c0b0 2024-03-23T15:54:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e291e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc6dc0cff10 con 0x7fc6dc1544b0 2024-03-23T15:54:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6e0ebb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 0x7fc6dc147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.535+0000 7fc6db7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc1544b0 0x7fc6dc146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.536+0000 7fc6e0ebb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 0x7fc6dc147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44210/0 (socket says 172.21.15.129:44210) 2024-03-23T15:54:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.536+0000 7fc6dbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6dc06c0b0 0x7fc6dc0d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.536+0000 7fc6e0ebb640 1 -- 172.21.15.129:0/2545730209 learned_addr learned my addr 172.21.15.129:0/2545730209 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:00.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.539+0000 7fc6e0ebb640 1 -- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc1544b0 msgr2=0x7fc6dc146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:00.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.539+0000 7fc6e0ebb640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc1544b0 0x7fc6dc146b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:00.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.539+0000 7fc6e0ebb640 1 -- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6dc06c0b0 msgr2=0x7fc6dc0d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:00.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.539+0000 7fc6e0ebb640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6dc06c0b0 0x7fc6dc0d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:00.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.539+0000 7fc6e0ebb640 1 -- 172.21.15.129:0/2545730209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d0009400 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.542+0000 7fc6e0ebb640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 0x7fc6dc147040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400efc0 tx=0x7fc6d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:00.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.546+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400ced0 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.546+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d400f040 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.546+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d40146c0 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.546+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6dc0d07b0 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.546+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6dc0cde80 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.547+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6dc146820 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.566+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc6d4014820 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.566+0000 7fc6d8ff9640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc6c806b810 0x7fc6c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.567+0000 7fc6dbfff640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc6c806b810 0x7fc6c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.567+0000 7fc6dbfff640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc6c806b810 0x7fc6c806dcd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007fd0 tx=0x7fc6cc007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:00.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.569+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc6d4059b80 con 0x7fc6dc15a2e0 2024-03-23T15:54:00.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:00.821+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6d4055e20 con 0x7fc6dc15a2e0 2024-03-23T15:54:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.049+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6dc146820 con 0x7fc6dc15a2e0 2024-03-23T15:54:01.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.080+0000 7fc6d8ff9640 1 -- 172.21.15.129:0/2545730209 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc6dc146820 con 0x7fc6dc15a2e0 2024-03-23T15:54:01.083 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:01.083 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:01.084 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:01.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc6c806b810 msgr2=0x7fc6c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc6c806b810 0x7fc6c806dcd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007fd0 tx=0x7fc6cc007e80 comp rx=0 tx=0).stop 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 msgr2=0x7fc6dc147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 0x7fc6dc147040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400efc0 tx=0x7fc6d400c490 comp rx=0 tx=0).stop 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 shutdown_connections 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6dc15a2e0 0x7fc6dc147040 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6dc1544b0 0x7fc6dc146b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc6c806b810 0x7fc6c806dcd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 --2- 172.21.15.129:0/2545730209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6dc06c0b0 0x7fc6dc0d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 >> 172.21.15.129:0/2545730209 conn(0x7fc6dc061b50 msgr2=0x7fc6dc0b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 shutdown_connections 2024-03-23T15:54:01.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:01.082+0000 7fc6e291e640 1 -- 172.21.15.129:0/2545730209 wait complete. 2024-03-23T15:54:01.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:01.701 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:02.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:02.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.801+0000 7fafcd4ae640 1 Processor -- start 2024-03-23T15:54:02.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.801+0000 7fafcd4ae640 1 -- start start 2024-03-23T15:54:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.802+0000 7fafcd4ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.802+0000 7fafcd4ae640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fafc80c6000 0x7fafc80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafcd4ae640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc80cbcb0 con 0x7fafc80caa00 2024-03-23T15:54:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafcd4ae640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fafc80c5be0 con 0x7fafc80c6000 2024-03-23T15:54:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafcd4ae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fafc80c69b0 con 0x7fafc81544b0 2024-03-23T15:54:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafc67fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fafc80caa00 0x7fafc806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:34940/0 (socket says 172.21.15.129:34940) 2024-03-23T15:54:02.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafc67fc640 1 -- 172.21.15.129:0/1157645559 learned_addr learned my addr 172.21.15.129:0/1157645559 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:02.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafc77fe640 1 --2- 172.21.15.129:0/1157645559 >> v2:172.21.15.157:3300/0 conn(0x7fafc80c6000 0x7fafc80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:02.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.803+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.806+0000 7fafc77fe640 1 -- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 msgr2=0x7fafc815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.806+0000 7fafc77fe640 1 --2- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.806+0000 7fafc77fe640 1 -- 172.21.15.129:0/1157645559 >> v1:172.21.15.129:6789/0 conn(0x7fafc80caa00 legacy=0x7fafc806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.806+0000 7fafc77fe640 1 -- 172.21.15.129:0/1157645559 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc815b280 con 0x7fafc80c6000 2024-03-23T15:54:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.808+0000 7fafc77fe640 1 --2- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80c6000 0x7fafc80c6470 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009d90 tx=0x7fafc002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a2bec208a873f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.810+0000 7fafc5ffb640 1 -- 172.21.15.129:0/1157645559 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc002cce0 con 0x7fafc80c6000 2024-03-23T15:54:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.812+0000 7fafc5ffb640 1 -- 172.21.15.129:0/1157645559 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc002ce40 con 0x7fafc80c6000 2024-03-23T15:54:02.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.812+0000 7fafc5ffb640 1 -- 172.21.15.129:0/1157645559 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0034660 con 0x7fafc80c6000 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 -- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80c6000 msgr2=0x7fafc80c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80c6000 0x7fafc80c6470 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009d90 tx=0x7fafc002bf30 comp rx=0 tx=0).stop 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 -- 172.21.15.129:0/1157645559 shutdown_connections 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80c6000 0x7fafc80c6470 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/1157645559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 -- 172.21.15.129:0/1157645559 >> 172.21.15.129:0/1157645559 conn(0x7fafc8061b50 msgr2=0x7fafc8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:02.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 -- 172.21.15.129:0/1157645559 shutdown_connections 2024-03-23T15:54:02.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.813+0000 7fafcd4ae640 1 -- 172.21.15.129:0/1157645559 wait complete. 2024-03-23T15:54:02.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.814+0000 7fafcd4ae640 1 Processor -- start 2024-03-23T15:54:02.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.814+0000 7fafcd4ae640 1 -- start start 2024-03-23T15:54:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.815+0000 7fafcd4ae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 0x7fafc8157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.815+0000 7fafcd4ae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80caa00 0x7fafc8155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.815+0000 7fafcd4ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.815+0000 7fafcd4ae640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8146f80 con 0x7fafc80c6000 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.815+0000 7fafcd4ae640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8145ba0 con 0x7fafc80caa00 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.815+0000 7fafcd4ae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fafc81463b0 con 0x7fafc81544b0 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.816+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 0x7fafc8157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.816+0000 7fafc67fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80caa00 0x7fafc8155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.816+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.816+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 0x7fafc8157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44224/0 (socket says 172.21.15.129:44224) 2024-03-23T15:54:02.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.816+0000 7fafc6ffd640 1 -- 172.21.15.129:0/2757768023 learned_addr learned my addr 172.21.15.129:0/2757768023 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:02.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.819+0000 7fafc6ffd640 1 -- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 msgr2=0x7fafc8155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:02.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.819+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc8155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:02.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.819+0000 7fafc6ffd640 1 -- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80caa00 msgr2=0x7fafc8155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:02.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.819+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80caa00 0x7fafc8155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:02.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.819+0000 7fafc6ffd640 1 -- 172.21.15.129:0/2757768023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc0009a70 con 0x7fafc80c6000 2024-03-23T15:54:02.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.821+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 0x7fafc8157420 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fafb800e870 tx=0x7fafb800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:02.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.827+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb80094f0 con 0x7fafc80c6000 2024-03-23T15:54:02.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.827+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb800c940 con 0x7fafc80c6000 2024-03-23T15:54:02.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.827+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc8157960 con 0x7fafc80c6000 2024-03-23T15:54:02.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.827+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb800ccf0 con 0x7fafc80c6000 2024-03-23T15:54:02.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.827+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc81460d0 con 0x7fafc80c6000 2024-03-23T15:54:02.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.828+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc8075740 con 0x7fafc80c6000 2024-03-23T15:54:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.859+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fafb8013470 con 0x7fafc80c6000 2024-03-23T15:54:02.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.859+0000 7fafaf7fe640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fafb406b860 0x7fafb406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:02.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.860+0000 7fafc67fc640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fafb406b860 0x7fafb406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:02.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.860+0000 7fafc67fc640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fafb406b860 0x7fafb406dd20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fafc81eda40 tx=0x7fafbc007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:02.862+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fafb8014e40 con 0x7fafc80c6000 2024-03-23T15:54:03.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.117+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fafb8055430 con 0x7fafc80c6000 2024-03-23T15:54:03.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.339+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafc8157d00 con 0x7fafc80c6000 2024-03-23T15:54:03.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.388+0000 7fafaf7fe640 1 -- 172.21.15.129:0/2757768023 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fafb8021090 con 0x7fafc80c6000 2024-03-23T15:54:03.392 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:03.392 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fafb406b860 msgr2=0x7fafb406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fafb406b860 0x7fafb406dd20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fafc81eda40 tx=0x7fafbc007e80 comp rx=0 tx=0).stop 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 msgr2=0x7fafc8157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 0x7fafc8157420 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fafb800e870 tx=0x7fafb800ed30 comp rx=0 tx=0).stop 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 shutdown_connections 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc81544b0 0x7fafc8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fafb406b860 0x7fafb406dd20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc80caa00 0x7fafc8155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 --2- 172.21.15.129:0/2757768023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc80c6000 0x7fafc8157420 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 >> 172.21.15.129:0/2757768023 conn(0x7fafc8061b50 msgr2=0x7fafc80cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 shutdown_connections 2024-03-23T15:54:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:03.391+0000 7fafcd4ae640 1 -- 172.21.15.129:0/2757768023 wait complete. 2024-03-23T15:54:04.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:04.013 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:05.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:05.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.112+0000 7fb3eefe2640 1 Processor -- start 2024-03-23T15:54:05.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.112+0000 7fb3eefe2640 1 -- start start 2024-03-23T15:54:05.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.113+0000 7fb3eefe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80ccd60 0x7fb3e815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.113+0000 7fb3eefe2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3e80ceb00 0x7fb3e806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.113+0000 7fb3eefe2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb3e80cef40 0x7fb3e806c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.113+0000 7fb3eefe2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb3e815b290 con 0x7fb3e80ceb00 2024-03-23T15:54:05.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.113+0000 7fb3eefe2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb3e80ce800 con 0x7fb3e80cef40 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.113+0000 7fb3eefe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e8154a70 con 0x7fb3e80ccd60 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.114+0000 7fb3e7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3e80ceb00 0x7fb3e806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.114+0000 7fb3ed57f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb3e80cef40 0x7fb3e806c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.114+0000 7fb3e7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3e80ceb00 0x7fb3e806c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44242/0 (socket says 172.21.15.129:44242) 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.114+0000 7fb3e7fff640 1 -- 172.21.15.129:0/1453384298 learned_addr learned my addr 172.21.15.129:0/1453384298 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.115+0000 7fb3ecd7e640 1 --2- 172.21.15.129:0/1453384298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80ccd60 0x7fb3e815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.116+0000 7fb3e7fff640 1 -- 172.21.15.129:0/1453384298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80ccd60 msgr2=0x7fb3e815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.116+0000 7fb3e7fff640 1 --2- 172.21.15.129:0/1453384298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80ccd60 0x7fb3e815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.116+0000 7fb3e7fff640 1 -- 172.21.15.129:0/1453384298 >> v2:172.21.15.157:3300/0 conn(0x7fb3e80cef40 msgr2=0x7fb3e806c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.116+0000 7fb3e7fff640 1 --2- 172.21.15.129:0/1453384298 >> v2:172.21.15.157:3300/0 conn(0x7fb3e80cef40 0x7fb3e806c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.116+0000 7fb3e7fff640 1 -- 172.21.15.129:0/1453384298 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e80cd950 con 0x7fb3e80ceb00 2024-03-23T15:54:05.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.118+0000 7fb3e7fff640 1 --2- 172.21.15.129:0/1453384298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e806c270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb3e815e8e0 tx=0x7fb3d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2d95b7accced65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:05.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.121+0000 7fb3e77fe640 1 -- 172.21.15.129:0/1453384298 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4030860 con 0x7fb3e80ceb00 2024-03-23T15:54:05.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.122+0000 7fb3e77fe640 1 -- 172.21.15.129:0/1453384298 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d40309c0 con 0x7fb3e80ceb00 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 -- 172.21.15.129:0/1453384298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 msgr2=0x7fb3e806c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/1453384298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e806c270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb3e815e8e0 tx=0x7fb3d402b9a0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 -- 172.21.15.129:0/1453384298 shutdown_connections 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/1453384298 >> v2:172.21.15.157:3300/0 conn(0x7fb3e80cef40 0x7fb3e806c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/1453384298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e806c270 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/1453384298 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80ccd60 0x7fb3e815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 -- 172.21.15.129:0/1453384298 >> 172.21.15.129:0/1453384298 conn(0x7fb3e80b1840 msgr2=0x7fb3e80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 -- 172.21.15.129:0/1453384298 shutdown_connections 2024-03-23T15:54:05.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.123+0000 7fb3eefe2640 1 -- 172.21.15.129:0/1453384298 wait complete. 2024-03-23T15:54:05.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.124+0000 7fb3eefe2640 1 Processor -- start 2024-03-23T15:54:05.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.124+0000 7fb3eefe2640 1 -- start start 2024-03-23T15:54:05.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.125+0000 7fb3eefe2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3e80ccd60 0x7fb3e81f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.125+0000 7fb3eefe2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e81e44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.125+0000 7fb3eefe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80cef40 0x7fb3e814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.125+0000 7fb3eefe2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8154c40 con 0x7fb3e80ceb00 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.125+0000 7fb3eefe2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81e4010 con 0x7fb3e80ccd60 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.125+0000 7fb3eefe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e81f3fd0 con 0x7fb3e80cef40 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.126+0000 7fb3e7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e81e44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.126+0000 7fb3ed57f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80cef40 0x7fb3e814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.126+0000 7fb3ecd7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3e80ccd60 0x7fb3e81f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.126+0000 7fb3e7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e81e44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44254/0 (socket says 172.21.15.129:44254) 2024-03-23T15:54:05.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.126+0000 7fb3e7fff640 1 -- 172.21.15.129:0/3214087720 learned_addr learned my addr 172.21.15.129:0/3214087720 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:05.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.129+0000 7fb3e7fff640 1 -- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80cef40 msgr2=0x7fb3e814a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:05.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.129+0000 7fb3e7fff640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80cef40 0x7fb3e814a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.129+0000 7fb3e7fff640 1 -- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3e80ccd60 msgr2=0x7fb3e81f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:05.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.129+0000 7fb3e7fff640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3e80ccd60 0x7fb3e81f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.129+0000 7fb3e7fff640 1 -- 172.21.15.129:0/3214087720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d40094d0 con 0x7fb3e80ceb00 2024-03-23T15:54:05.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.132+0000 7fb3e7fff640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e81e44a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb3d402b9e0 tx=0x7fb3d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:05.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.135+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4003810 con 0x7fb3e80ceb00 2024-03-23T15:54:05.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.135+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d4003990 con 0x7fb3e80ceb00 2024-03-23T15:54:05.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.135+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4003d40 con 0x7fb3e80ceb00 2024-03-23T15:54:05.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.135+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e81f4290 con 0x7fb3e80ceb00 2024-03-23T15:54:05.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.136+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81e39c0 con 0x7fb3e80ceb00 2024-03-23T15:54:05.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.137+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e804f530 con 0x7fb3e80ceb00 2024-03-23T15:54:05.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.156+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb3d404d450 con 0x7fb3e80ceb00 2024-03-23T15:54:05.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.157+0000 7fb3e57fa640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb3cc06b890 0x7fb3cc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:05.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.157+0000 7fb3ecd7e640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb3cc06b890 0x7fb3cc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:05.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.157+0000 7fb3ecd7e640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb3cc06b890 0x7fb3cc06dd50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0079a0 tx=0x7fb3dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:05.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.160+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb3d407b3d0 con 0x7fb3e80ceb00 2024-03-23T15:54:05.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.439+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb3d4077670 con 0x7fb3e80ceb00 2024-03-23T15:54:05.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.665+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3e806db50 con 0x7fb3e80ceb00 2024-03-23T15:54:05.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.679+0000 7fb3e57fa640 1 -- 172.21.15.129:0/3214087720 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb3d407ad30 con 0x7fb3e80ceb00 2024-03-23T15:54:05.682 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:05.682 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:05.682 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb3cc06b890 msgr2=0x7fb3cc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb3cc06b890 0x7fb3cc06dd50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0079a0 tx=0x7fb3dc008040 comp rx=0 tx=0).stop 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 msgr2=0x7fb3e81e44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e81e44a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb3d402b9e0 tx=0x7fb3d40046c0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 shutdown_connections 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3e80cef40 0x7fb3e814a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3e80ceb00 0x7fb3e81e44a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb3cc06b890 0x7fb3cc06dd50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 --2- 172.21.15.129:0/3214087720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3e80ccd60 0x7fb3e81f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:05.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 >> 172.21.15.129:0/3214087720 conn(0x7fb3e80b1840 msgr2=0x7fb3e815a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:05.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 shutdown_connections 2024-03-23T15:54:05.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:05.681+0000 7fb3eefe2640 1 -- 172.21.15.129:0/3214087720 wait complete. 2024-03-23T15:54:06.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:06.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:07.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.401+0000 7f9d0646d640 1 Processor -- start 2024-03-23T15:54:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.401+0000 7f9d0646d640 1 -- start start 2024-03-23T15:54:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.402+0000 7f9d0646d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9d000ceb00 0x7f9d0015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.402+0000 7f9d0646d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000cef40 0x7f9d0015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.402+0000 7f9d0646d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d0015b290 con 0x7f9d000ccd60 2024-03-23T15:54:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.402+0000 7f9d0646d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9d000ce800 con 0x7f9d000ceb00 2024-03-23T15:54:07.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.402+0000 7f9d0646d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9d00154a70 con 0x7f9d000cef40 2024-03-23T15:54:07.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.403+0000 7f9d04a0a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000cef40 0x7f9d0015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:07.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.403+0000 7f9cff7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9d000ceb00 0x7f9d0015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:07.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.403+0000 7f9cfffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9d000ccd60 0x7f9d0006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43164/0 (socket says 172.21.15.129:43164) 2024-03-23T15:54:07.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.404+0000 7f9cfffff640 1 -- 172.21.15.129:0/1878941462 learned_addr learned my addr 172.21.15.129:0/1878941462 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:07.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.406+0000 7f9d04a0a640 1 -- 172.21.15.129:0/1878941462 >> v2:172.21.15.157:3300/0 conn(0x7f9d000ceb00 msgr2=0x7f9d0015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:07.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.406+0000 7f9d04a0a640 1 --2- 172.21.15.129:0/1878941462 >> v2:172.21.15.157:3300/0 conn(0x7f9d000ceb00 0x7f9d0015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:07.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.406+0000 7f9d04a0a640 1 -- 172.21.15.129:0/1878941462 >> v1:172.21.15.157:6789/0 conn(0x7f9d000ccd60 legacy=0x7f9d0006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:07.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.406+0000 7f9d04a0a640 1 -- 172.21.15.129:0/1878941462 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d000cd950 con 0x7f9d000cef40 2024-03-23T15:54:07.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.408+0000 7f9d04a0a640 1 --2- 172.21.15.129:0/1878941462 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000cef40 0x7f9d0015b8f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009d90 tx=0x7f9cf802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de568c2f0ec448c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.410+0000 7f9cfeffd640 1 -- 172.21.15.129:0/1878941462 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf802cce0 con 0x7f9d000cef40 2024-03-23T15:54:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.413+0000 7f9cfeffd640 1 -- 172.21.15.129:0/1878941462 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf802ce40 con 0x7f9d000cef40 2024-03-23T15:54:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.413+0000 7f9cfeffd640 1 -- 172.21.15.129:0/1878941462 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8034660 con 0x7f9d000cef40 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 -- 172.21.15.129:0/1878941462 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000cef40 msgr2=0x7f9d0015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 --2- 172.21.15.129:0/1878941462 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000cef40 0x7f9d0015b8f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009d90 tx=0x7f9cf802bf30 comp rx=0 tx=0).stop 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 -- 172.21.15.129:0/1878941462 shutdown_connections 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 --2- 172.21.15.129:0/1878941462 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000cef40 0x7f9d0015b8f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 --2- 172.21.15.129:0/1878941462 >> v2:172.21.15.157:3300/0 conn(0x7f9d000ceb00 0x7f9d0015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 -- 172.21.15.129:0/1878941462 >> 172.21.15.129:0/1878941462 conn(0x7f9d000b1840 msgr2=0x7f9d000b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 -- 172.21.15.129:0/1878941462 shutdown_connections 2024-03-23T15:54:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.414+0000 7f9d0646d640 1 -- 172.21.15.129:0/1878941462 wait complete. 2024-03-23T15:54:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.415+0000 7f9d0646d640 1 Processor -- start 2024-03-23T15:54:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.415+0000 7f9d0646d640 1 -- start start 2024-03-23T15:54:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9d0646d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000ccd60 0x7f9d001e4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9d0646d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 0x7f9d00151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9d0646d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d000cef40 0x7f9d001522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9d0646d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0015a670 con 0x7f9d000cef40 2024-03-23T15:54:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9d0646d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0015a390 con 0x7f9d000ceb00 2024-03-23T15:54:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9d0646d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9d001567d0 con 0x7f9d000ccd60 2024-03-23T15:54:07.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9cff7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 0x7f9d00151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:07.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.416+0000 7f9cfffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000ccd60 0x7f9d001e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:07.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.417+0000 7f9cff7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 0x7f9d00151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52062/0 (socket says 172.21.15.129:52062) 2024-03-23T15:54:07.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.417+0000 7f9cff7fe640 1 -- 172.21.15.129:0/3105340709 learned_addr learned my addr 172.21.15.129:0/3105340709 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:07.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.417+0000 7f9d04a0a640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d000cef40 0x7f9d001522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.420+0000 7f9cff7fe640 1 -- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000ccd60 msgr2=0x7f9d001e4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.420+0000 7f9cff7fe640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000ccd60 0x7f9d001e4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.420+0000 7f9cff7fe640 1 -- 172.21.15.129:0/3105340709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d000cef40 msgr2=0x7f9d001522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.420+0000 7f9cff7fe640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d000cef40 0x7f9d001522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.420+0000 7f9cff7fe640 1 -- 172.21.15.129:0/3105340709 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf8009a70 con 0x7f9d000ceb00 2024-03-23T15:54:07.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.423+0000 7f9cff7fe640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 0x7f9d00151d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e870 tx=0x7f9cf400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.427+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf40094f0 con 0x7f9d000ceb00 2024-03-23T15:54:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.428+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf400c940 con 0x7f9d000ceb00 2024-03-23T15:54:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.428+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400ccf0 con 0x7f9d000ceb00 2024-03-23T15:54:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.428+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d00156ab0 con 0x7f9d000ceb00 2024-03-23T15:54:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.428+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d00157070 con 0x7f9d000ceb00 2024-03-23T15:54:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.429+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d00157230 con 0x7f9d000ceb00 2024-03-23T15:54:07.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:54:07.601 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:54:07.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.705+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9cf4013470 con 0x7f9d000ceb00 2024-03-23T15:54:07.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.705+0000 7f9cfcff9640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cec06b860 0x7f9cec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:07.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.706+0000 7f9cfffff640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cec06b860 0x7f9cec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:07.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.706+0000 7f9cfffff640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cec06b860 0x7f9cec06dd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0005880 tx=0x7f9cf0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:07.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.713+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9cf4014e40 con 0x7f9d000ceb00 2024-03-23T15:54:07.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.732+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9cf4055430 con 0x7f9d000ceb00 2024-03-23T15:54:07.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.970+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d0006db50 con 0x7f9d000ceb00 2024-03-23T15:54:07.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.994+0000 7f9cfcff9640 1 -- 172.21.15.129:0/3105340709 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9cf4021090 con 0x7f9d000ceb00 2024-03-23T15:54:07.997 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:07.997 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:07.998 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:08.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cec06b860 msgr2=0x7f9cec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:08.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cec06b860 0x7f9cec06dd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0005880 tx=0x7f9cf0009040 comp rx=0 tx=0).stop 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 msgr2=0x7f9d00151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 0x7f9d00151d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e870 tx=0x7f9cf400ed30 comp rx=0 tx=0).stop 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 shutdown_connections 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d000cef40 0x7f9d001522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9cec06b860 0x7f9cec06dd20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d000ceb00 0x7f9d00151d70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 --2- 172.21.15.129:0/3105340709 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d000ccd60 0x7f9d001e4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 >> 172.21.15.129:0/3105340709 conn(0x7f9d000b1840 msgr2=0x7f9d00153fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 shutdown_connections 2024-03-23T15:54:08.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:07.997+0000 7f9d0646d640 1 -- 172.21.15.129:0/3105340709 wait complete. 2024-03-23T15:54:08.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:08.616 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:09.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:09.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.716+0000 7faee1c15640 1 Processor -- start 2024-03-23T15:54:09.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.716+0000 7faee1c15640 1 -- start start 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.717+0000 7faee1c15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc06c410 0x7faedc15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.717+0000 7faee1c15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faedc06c830 con 0x7faedc15de40 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.717+0000 7faee1c15640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faedc0cc370 con 0x7faedc06c030 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.717+0000 7faee1c15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faedc06cae0 con 0x7faedc06c410 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.718+0000 7faedbfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7faedc15de40 0x7faedc164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:34950/0 (socket says 172.21.15.129:34950) 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.718+0000 7faedbfff640 1 -- 172.21.15.129:0/1057998911 learned_addr learned my addr 172.21.15.129:0/1057998911 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:09.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.718+0000 7faedaffd640 1 --2- 172.21.15.129:0/1057998911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc06c410 0x7faedc15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:09.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.720+0000 7faeda7fc640 1 -- 172.21.15.129:0/1057998911 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808784757 0 0) 0x7faedc06c830 con 0x7faedc15de40 2024-03-23T15:54:09.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.721+0000 7faeda7fc640 1 -- 172.21.15.129:0/1057998911 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faec4003520 con 0x7faedc15de40 2024-03-23T15:54:09.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.721+0000 7faedaffd640 1 -- 172.21.15.129:0/1057998911 >> v1:172.21.15.157:6789/0 conn(0x7faedc06c030 legacy=0x7faedc0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:09.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.721+0000 7faedaffd640 1 -- 172.21.15.129:0/1057998911 >> v1:172.21.15.129:6789/0 conn(0x7faedc15de40 legacy=0x7faedc164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:09.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.721+0000 7faedaffd640 1 -- 172.21.15.129:0/1057998911 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faedc0cabc0 con 0x7faedc06c410 2024-03-23T15:54:09.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.724+0000 7faedaffd640 1 --2- 172.21.15.129:0/1057998911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc06c410 0x7faedc15d900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faed00097d0 tx=0x7faed002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f86fb9a5fc4f6cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:09.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.729+0000 7faeda7fc640 1 -- 172.21.15.129:0/1057998911 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0031890 con 0x7faedc06c410 2024-03-23T15:54:09.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.729+0000 7faeda7fc640 1 -- 172.21.15.129:0/1057998911 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed00319f0 con 0x7faedc06c410 2024-03-23T15:54:09.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.729+0000 7faeda7fc640 1 -- 172.21.15.129:0/1057998911 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0031d80 con 0x7faedc06c410 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 -- 172.21.15.129:0/1057998911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc06c410 msgr2=0x7faedc15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 --2- 172.21.15.129:0/1057998911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc06c410 0x7faedc15d900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faed00097d0 tx=0x7faed002b930 comp rx=0 tx=0).stop 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 -- 172.21.15.129:0/1057998911 shutdown_connections 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 --2- 172.21.15.129:0/1057998911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc06c410 0x7faedc15d900 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 -- 172.21.15.129:0/1057998911 >> 172.21.15.129:0/1057998911 conn(0x7faedc0cfb20 msgr2=0x7faedc0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 -- 172.21.15.129:0/1057998911 shutdown_connections 2024-03-23T15:54:09.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.730+0000 7faee1c15640 1 -- 172.21.15.129:0/1057998911 wait complete. 2024-03-23T15:54:09.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.731+0000 7faee1c15640 1 Processor -- start 2024-03-23T15:54:09.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.731+0000 7faee1c15640 1 -- start start 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faee1c15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc06c030 0x7faedc150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faee1c15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 0x7faedc154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faee1c15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc15de40 0x7faedc1554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faee1c15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faedc1f0f10 con 0x7faedc06c410 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faee1c15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faedc15b270 con 0x7faedc06c030 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faee1c15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faedc0b54f0 con 0x7faedc15de40 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faedaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 0x7faedc154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:09.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.732+0000 7faedbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc15de40 0x7faedc1554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:09.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.733+0000 7faedb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc06c030 0x7faedc150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:09.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.733+0000 7faedaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 0x7faedc154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44276/0 (socket says 172.21.15.129:44276) 2024-03-23T15:54:09.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.733+0000 7faedaffd640 1 -- 172.21.15.129:0/2725549380 learned_addr learned my addr 172.21.15.129:0/2725549380 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:09.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.736+0000 7faedaffd640 1 -- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc15de40 msgr2=0x7faedc1554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:09.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.736+0000 7faedaffd640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc15de40 0x7faedc1554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:09.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.736+0000 7faedaffd640 1 -- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc06c030 msgr2=0x7faedc150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:09.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.736+0000 7faedaffd640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc06c030 0x7faedc150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:09.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.736+0000 7faedaffd640 1 -- 172.21.15.129:0/2725549380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed0009400 con 0x7faedc06c410 2024-03-23T15:54:09.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.738+0000 7faedaffd640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 0x7faedc154fa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faed00098f0 tx=0x7faed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:09.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.741+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed003bd80 con 0x7faedc06c410 2024-03-23T15:54:09.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.741+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed00453f0 con 0x7faedc06c410 2024-03-23T15:54:09.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.741+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faedc0b5790 con 0x7faedc06c410 2024-03-23T15:54:09.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.741+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed00457a0 con 0x7faedc06c410 2024-03-23T15:54:09.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.741+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faedc15af00 con 0x7faedc06c410 2024-03-23T15:54:09.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.742+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faedc0686f0 con 0x7faedc06c410 2024-03-23T15:54:09.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.762+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faed004e450 con 0x7faedc06c410 2024-03-23T15:54:09.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.762+0000 7faec3fff640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faec806b890 0x7faec806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:09.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.763+0000 7faedb7fe640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faec806b890 0x7faec806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:09.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:09.763+0000 7faedb7fe640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faec806b890 0x7faec806dd50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faecc000f60 tx=0x7faecc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.009+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7faed007b010 con 0x7faedc06c410 2024-03-23T15:54:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.009+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faed00772b0 con 0x7faedc06c410 2024-03-23T15:54:10.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.230+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faedc15b090 con 0x7faedc06c410 2024-03-23T15:54:10.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.244+0000 7faec3fff640 1 -- 172.21.15.129:0/2725549380 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7faed0035070 con 0x7faedc06c410 2024-03-23T15:54:10.247 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:10.247 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:10.248 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faec806b890 msgr2=0x7faec806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faec806b890 0x7faec806dd50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faecc000f60 tx=0x7faecc009040 comp rx=0 tx=0).stop 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 msgr2=0x7faedc154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 0x7faedc154fa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faed00098f0 tx=0x7faed00046c0 comp rx=0 tx=0).stop 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 shutdown_connections 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faedc15de40 0x7faedc1554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faedc06c410 0x7faedc154fa0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faec806b890 0x7faec806dd50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 --2- 172.21.15.129:0/2725549380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc06c030 0x7faedc150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:10.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.246+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 >> 172.21.15.129:0/2725549380 conn(0x7faedc0cfb20 msgr2=0x7faedc0c89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:10.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.248+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 shutdown_connections 2024-03-23T15:54:10.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:10.248+0000 7faee1c15640 1 -- 172.21.15.129:0/2725549380 wait complete. 2024-03-23T15:54:10.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:10.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:11.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:11.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.968+0000 7fe582674640 1 Processor -- start 2024-03-23T15:54:11.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.968+0000 7fe582674640 1 -- start start 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe582674640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe57c1544d0 0x7fe57c0c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe582674640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c0ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe582674640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57c15aa90 con 0x7fe57c15a200 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe582674640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe57c15ad70 con 0x7fe57c1544d0 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe582674640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c0c9600 con 0x7fe57c06b9a0 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe580c11640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fe57c15a200 0x7fe57c0ccb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:54576/0 (socket says 172.21.15.129:54576) 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe580c11640 1 -- 172.21.15.129:0/1119351451 learned_addr learned my addr 172.21.15.129:0/1119351451 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.969+0000 7fe57b7fe640 1 --2- 172.21.15.129:0/1119351451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c0ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.970+0000 7fe57bfff640 1 --2- 172.21.15.129:0/1119351451 >> v2:172.21.15.157:3300/0 conn(0x7fe57c1544d0 0x7fe57c0c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:11.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.972+0000 7fe57b7fe640 1 -- 172.21.15.129:0/1119351451 >> v2:172.21.15.157:3300/0 conn(0x7fe57c1544d0 msgr2=0x7fe57c0c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:11.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.972+0000 7fe57b7fe640 1 --2- 172.21.15.129:0/1119351451 >> v2:172.21.15.157:3300/0 conn(0x7fe57c1544d0 0x7fe57c0c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:11.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.972+0000 7fe57b7fe640 1 -- 172.21.15.129:0/1119351451 >> v1:172.21.15.129:6789/0 conn(0x7fe57c15a200 legacy=0x7fe57c0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:11.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.972+0000 7fe57b7fe640 1 -- 172.21.15.129:0/1119351451 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c0c98e0 con 0x7fe57c06b9a0 2024-03-23T15:54:11.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.974+0000 7fe57b7fe640 1 --2- 172.21.15.129:0/1119351451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c0ca0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe5700097a0 tx=0x7fe57002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e4cbeb647c84e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:11.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.983+0000 7fe57affd640 1 -- 172.21.15.129:0/1119351451 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570030610 con 0x7fe57c06b9a0 2024-03-23T15:54:11.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.983+0000 7fe57affd640 1 -- 172.21.15.129:0/1119351451 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe570030770 con 0x7fe57c06b9a0 2024-03-23T15:54:11.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.983+0000 7fe57affd640 1 -- 172.21.15.129:0/1119351451 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570030b00 con 0x7fe57c06b9a0 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.983+0000 7fe582674640 1 -- 172.21.15.129:0/1119351451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 msgr2=0x7fe57c0ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.983+0000 7fe582674640 1 --2- 172.21.15.129:0/1119351451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c0ca0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe5700097a0 tx=0x7fe57002b960 comp rx=0 tx=0).stop 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.984+0000 7fe582674640 1 -- 172.21.15.129:0/1119351451 shutdown_connections 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.984+0000 7fe582674640 1 --2- 172.21.15.129:0/1119351451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c0ca0e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.984+0000 7fe582674640 1 --2- 172.21.15.129:0/1119351451 >> v2:172.21.15.157:3300/0 conn(0x7fe57c1544d0 0x7fe57c0c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.984+0000 7fe582674640 1 -- 172.21.15.129:0/1119351451 >> 172.21.15.129:0/1119351451 conn(0x7fe57c06e230 msgr2=0x7fe57c06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.984+0000 7fe582674640 1 -- 172.21.15.129:0/1119351451 shutdown_connections 2024-03-23T15:54:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.984+0000 7fe582674640 1 -- 172.21.15.129:0/1119351451 wait complete. 2024-03-23T15:54:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.985+0000 7fe582674640 1 Processor -- start 2024-03-23T15:54:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.985+0000 7fe582674640 1 -- start start 2024-03-23T15:54:11.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe582674640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:11.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe582674640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 0x7fe57c156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe582674640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe57c15a200 0x7fe57c1f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe582674640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c155cb0 con 0x7fe57c1544d0 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe582674640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c157a90 con 0x7fe57c15a200 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe582674640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c1577d0 con 0x7fe57c06b9a0 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe57b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 0x7fe57c156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.986+0000 7fe580c11640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe57c15a200 0x7fe57c1f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.987+0000 7fe580c11640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe57c15a200 0x7fe57c1f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:56250/0 (socket says 172.21.15.129:56250) 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.987+0000 7fe57b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 0x7fe57c156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44748/0 (socket says 172.21.15.129:44748) 2024-03-23T15:54:11.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.987+0000 7fe580c11640 1 -- 172.21.15.129:0/534081364 learned_addr learned my addr 172.21.15.129:0/534081364 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:11.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.987+0000 7fe57bfff640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:11.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.992+0000 7fe57b7fe640 1 -- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 msgr2=0x7fe57c153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:11.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.992+0000 7fe57b7fe640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:11.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.992+0000 7fe57b7fe640 1 -- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe57c15a200 msgr2=0x7fe57c1f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:11.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.992+0000 7fe57b7fe640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe57c15a200 0x7fe57c1f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:11.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.992+0000 7fe57b7fe640 1 -- 172.21.15.129:0/534081364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe570009400 con 0x7fe57c1544d0 2024-03-23T15:54:11.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:11.994+0000 7fe57b7fe640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 0x7fe57c156e00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe570033000 tx=0x7fe570004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:12.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.000+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57003e040 con 0x7fe57c1544d0 2024-03-23T15:54:12.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.000+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe570039a40 con 0x7fe57c1544d0 2024-03-23T15:54:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.000+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c15bc20 con 0x7fe57c1544d0 2024-03-23T15:54:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.000+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570039df0 con 0x7fe57c1544d0 2024-03-23T15:54:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.000+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c15c1e0 con 0x7fe57c1544d0 2024-03-23T15:54:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.001+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe57c15c700 con 0x7fe57c1544d0 2024-03-23T15:54:12.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.273+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe57004d3b0 con 0x7fe57c1544d0 2024-03-23T15:54:12.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.273+0000 7fe578ff9640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe56806b860 0x7fe56806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:12.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.274+0000 7fe57bfff640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe56806b860 0x7fe56806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:12.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.274+0000 7fe57bfff640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe56806b860 0x7fe56806dd20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009b40 tx=0x7fe56c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:12.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.295+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe57007b1b0 con 0x7fe57c1544d0 2024-03-23T15:54:12.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.295+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe570077450 con 0x7fe57c1544d0 2024-03-23T15:54:12.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.545+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe57c15c700 con 0x7fe57c1544d0 2024-03-23T15:54:12.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.559+0000 7fe578ff9640 1 -- 172.21.15.129:0/534081364 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe57c15c700 con 0x7fe57c1544d0 2024-03-23T15:54:12.563 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:12.563 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:12.563 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:12.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe56806b860 msgr2=0x7fe56806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe56806b860 0x7fe56806dd20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009b40 tx=0x7fe56c009190 comp rx=0 tx=0).stop 2024-03-23T15:54:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 msgr2=0x7fe57c156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 0x7fe57c156e00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe570033000 tx=0x7fe570004730 comp rx=0 tx=0).stop 2024-03-23T15:54:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 shutdown_connections 2024-03-23T15:54:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe56806b860 0x7fe56806dd20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe57c15a200 0x7fe57c1f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe57c1544d0 0x7fe57c156e00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 --2- 172.21.15.129:0/534081364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe57c06b9a0 0x7fe57c153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 >> 172.21.15.129:0/534081364 conn(0x7fe57c06e230 msgr2=0x7fe57c0c7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 shutdown_connections 2024-03-23T15:54:12.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:12.562+0000 7fe582674640 1 -- 172.21.15.129:0/534081364 wait complete. 2024-03-23T15:54:13.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:13.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:14.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:14.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.283+0000 7f0327a64640 1 Processor -- start 2024-03-23T15:54:14.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.283+0000 7f0327a64640 1 -- start start 2024-03-23T15:54:14.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.284+0000 7f0327a64640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f03201544b0 0x7f032015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:14.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.284+0000 7f0327a64640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200cbcb0 con 0x7f03200caa00 2024-03-23T15:54:14.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.284+0000 7f0327a64640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032015b280 con 0x7f03200c6000 2024-03-23T15:54:14.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.284+0000 7f0327a64640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f03200c5be0 con 0x7f03201544b0 2024-03-23T15:54:14.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.285+0000 7f0324fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f03200caa00 0x7f032006bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:54588/0 (socket says 172.21.15.129:54588) 2024-03-23T15:54:14.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.285+0000 7f0324fff640 1 -- 172.21.15.129:0/245266265 learned_addr learned my addr 172.21.15.129:0/245266265 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:14.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.285+0000 7f0325800640 1 --2- 172.21.15.129:0/245266265 >> v2:172.21.15.157:3300/0 conn(0x7f03201544b0 0x7f032015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:14.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.288+0000 7f0317fff640 1 -- 172.21.15.129:0/245266265 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997797591 0 0) 0x7f03200cbcb0 con 0x7f03200caa00 2024-03-23T15:54:14.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.288+0000 7f0317fff640 1 -- 172.21.15.129:0/245266265 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f030c003520 con 0x7f03200caa00 2024-03-23T15:54:14.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.290+0000 7f0325800640 1 -- 172.21.15.129:0/245266265 >> v1:172.21.15.157:6789/0 conn(0x7f03200c6000 legacy=0x7f0320164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.290+0000 7f0325800640 1 -- 172.21.15.129:0/245266265 >> v1:172.21.15.129:6789/0 conn(0x7f03200caa00 legacy=0x7f032006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.290+0000 7f0325800640 1 -- 172.21.15.129:0/245266265 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032015a9e0 con 0x7f03201544b0 2024-03-23T15:54:14.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.295+0000 7f0325800640 1 --2- 172.21.15.129:0/245266265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03201544b0 0x7f032015a2e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f031c0097a0 tx=0x7f031c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab93d81240506863 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:14.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0317fff640 1 -- 172.21.15.129:0/245266265 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c02f810 con 0x7f03201544b0 2024-03-23T15:54:14.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0317fff640 1 -- 172.21.15.129:0/245266265 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031c02f970 con 0x7f03201544b0 2024-03-23T15:54:14.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0317fff640 1 -- 172.21.15.129:0/245266265 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c02fd00 con 0x7f03201544b0 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0327a64640 1 -- 172.21.15.129:0/245266265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03201544b0 msgr2=0x7f032015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0327a64640 1 --2- 172.21.15.129:0/245266265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03201544b0 0x7f032015a2e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f031c0097a0 tx=0x7f031c02b9f0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0327a64640 1 -- 172.21.15.129:0/245266265 shutdown_connections 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0327a64640 1 --2- 172.21.15.129:0/245266265 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03201544b0 0x7f032015a2e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0327a64640 1 -- 172.21.15.129:0/245266265 >> 172.21.15.129:0/245266265 conn(0x7f0320061b50 msgr2=0x7f0320061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.301+0000 7f0327a64640 1 -- 172.21.15.129:0/245266265 shutdown_connections 2024-03-23T15:54:14.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.302+0000 7f0327a64640 1 -- 172.21.15.129:0/245266265 wait complete. 2024-03-23T15:54:14.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 Processor -- start 2024-03-23T15:54:14.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 -- start start 2024-03-23T15:54:14.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03200c6000 0x7f03200c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03200caa00 0x7f032014d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201544b0 0x7f03200cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0320157770 con 0x7f03201544b0 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03200c75a0 con 0x7f03200caa00 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.303+0000 7f0327a64640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0320156e80 con 0x7f03200c6000 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.304+0000 7f0326001640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201544b0 0x7f03200cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:14.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.304+0000 7f0324fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03200caa00 0x7f032014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:14.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.304+0000 7f0325800640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03200c6000 0x7f03200c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:14.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.304+0000 7f0324fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03200caa00 0x7f032014d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:56266/0 (socket says 172.21.15.129:56266) 2024-03-23T15:54:14.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.304+0000 7f0324fff640 1 -- 172.21.15.129:0/3458356832 learned_addr learned my addr 172.21.15.129:0/3458356832 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:14.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.308+0000 7f0326001640 1 -- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03200c6000 msgr2=0x7f03200c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.308+0000 7f0326001640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03200c6000 0x7f03200c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.308+0000 7f0326001640 1 -- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03200caa00 msgr2=0x7f032014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.308+0000 7f0326001640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03200caa00 0x7f032014d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.308+0000 7f0326001640 1 -- 172.21.15.129:0/3458356832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031c009400 con 0x7f03201544b0 2024-03-23T15:54:14.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.310+0000 7f0326001640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201544b0 0x7f03200cef80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f031800c690 tx=0x7f031800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:14.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.312+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318003b70 con 0x7f03201544b0 2024-03-23T15:54:14.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.312+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03200c9ec0 con 0x7f03201544b0 2024-03-23T15:54:14.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.312+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0320157400 con 0x7f03201544b0 2024-03-23T15:54:14.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.313+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03201578e0 con 0x7f03201544b0 2024-03-23T15:54:14.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.315+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0318001780 con 0x7f03201544b0 2024-03-23T15:54:14.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.315+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180050d0 con 0x7f03201544b0 2024-03-23T15:54:14.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.591+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f03180052f0 con 0x7f03201544b0 2024-03-23T15:54:14.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.592+0000 7f0315ffb640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f030406b810 0x7f030406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:14.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.592+0000 7f0325800640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f030406b810 0x7f030406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:14.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.593+0000 7f0325800640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f030406b810 0x7f030406dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f031c036000 tx=0x7f031c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:14.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.606+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f03180591a0 con 0x7f03201544b0 2024-03-23T15:54:14.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.606+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0318058f70 con 0x7f03201544b0 2024-03-23T15:54:14.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.835+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0320157590 con 0x7f03201544b0 2024-03-23T15:54:14.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.855+0000 7f0315ffb640 1 -- 172.21.15.129:0/3458356832 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0318058b00 con 0x7f03201544b0 2024-03-23T15:54:14.858 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:14.858 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:14.858 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:14.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f030406b810 msgr2=0x7f030406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f030406b810 0x7f030406dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f031c036000 tx=0x7f031c008030 comp rx=0 tx=0).stop 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201544b0 msgr2=0x7f03200cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201544b0 0x7f03200cef80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f031800c690 tx=0x7f031800cb50 comp rx=0 tx=0).stop 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 shutdown_connections 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03201544b0 0x7f03200cef80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f030406b810 0x7f030406dcd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03200caa00 0x7f032014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 --2- 172.21.15.129:0/3458356832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03200c6000 0x7f03200c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 >> 172.21.15.129:0/3458356832 conn(0x7f0320061b50 msgr2=0x7f03200c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 shutdown_connections 2024-03-23T15:54:14.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:14.857+0000 7f0327a64640 1 -- 172.21.15.129:0/3458356832 wait complete. 2024-03-23T15:54:15.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:15.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:16.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.577+0000 7efd498c0640 1 Processor -- start 2024-03-23T15:54:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.577+0000 7efd498c0640 1 -- start start 2024-03-23T15:54:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.578+0000 7efd498c0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efd441544b0 0x7efd4415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.578+0000 7efd498c0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd440cbcb0 con 0x7efd440caa00 2024-03-23T15:54:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.578+0000 7efd498c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd4415b280 con 0x7efd440c6000 2024-03-23T15:54:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.578+0000 7efd498c0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efd440c5be0 con 0x7efd441544b0 2024-03-23T15:54:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.578+0000 7efd427fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7efd440caa00 0x7efd4406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:54592/0 (socket says 172.21.15.129:54592) 2024-03-23T15:54:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.578+0000 7efd427fc640 1 -- 172.21.15.129:0/3081668041 learned_addr learned my addr 172.21.15.129:0/3081668041 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:16.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.579+0000 7efd42ffd640 1 --2- 172.21.15.129:0/3081668041 >> v2:172.21.15.157:3300/0 conn(0x7efd441544b0 0x7efd4415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:16.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.585+0000 7efd41ffb640 1 -- 172.21.15.129:0/3081668041 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834829702 0 0) 0x7efd440cbcb0 con 0x7efd440caa00 2024-03-23T15:54:16.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.585+0000 7efd41ffb640 1 -- 172.21.15.129:0/3081668041 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd2c003520 con 0x7efd440caa00 2024-03-23T15:54:16.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.585+0000 7efd42ffd640 1 -- 172.21.15.129:0/3081668041 >> v1:172.21.15.157:6789/0 conn(0x7efd440c6000 legacy=0x7efd44164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:16.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.585+0000 7efd42ffd640 1 -- 172.21.15.129:0/3081668041 >> v1:172.21.15.129:6789/0 conn(0x7efd440caa00 legacy=0x7efd4406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:16.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.585+0000 7efd42ffd640 1 -- 172.21.15.129:0/3081668041 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4415a9e0 con 0x7efd441544b0 2024-03-23T15:54:16.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.593+0000 7efd42ffd640 1 --2- 172.21.15.129:0/3081668041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd441544b0 0x7efd4415a2e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efd340097a0 tx=0x7efd3402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c8928adaf8d9b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:16.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.598+0000 7efd41ffb640 1 -- 172.21.15.129:0/3081668041 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3402f810 con 0x7efd441544b0 2024-03-23T15:54:16.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.598+0000 7efd41ffb640 1 -- 172.21.15.129:0/3081668041 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3402f970 con 0x7efd441544b0 2024-03-23T15:54:16.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.598+0000 7efd41ffb640 1 -- 172.21.15.129:0/3081668041 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3402fd00 con 0x7efd441544b0 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 -- 172.21.15.129:0/3081668041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd441544b0 msgr2=0x7efd4415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 --2- 172.21.15.129:0/3081668041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd441544b0 0x7efd4415a2e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efd340097a0 tx=0x7efd3402b9f0 comp rx=0 tx=0).stop 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 -- 172.21.15.129:0/3081668041 shutdown_connections 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 --2- 172.21.15.129:0/3081668041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd441544b0 0x7efd4415a2e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 -- 172.21.15.129:0/3081668041 >> 172.21.15.129:0/3081668041 conn(0x7efd44061b50 msgr2=0x7efd44061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 -- 172.21.15.129:0/3081668041 shutdown_connections 2024-03-23T15:54:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.599+0000 7efd498c0640 1 -- 172.21.15.129:0/3081668041 wait complete. 2024-03-23T15:54:16.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.600+0000 7efd498c0640 1 Processor -- start 2024-03-23T15:54:16.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.600+0000 7efd498c0640 1 -- start start 2024-03-23T15:54:16.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd498c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd440c6000 0x7efd440c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:16.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd498c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd440caa00 0x7efd4414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:16.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd498c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd441544b0 0x7efd440cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:16.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd498c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd44157770 con 0x7efd440caa00 2024-03-23T15:54:16.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd498c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efd440c75a0 con 0x7efd440c6000 2024-03-23T15:54:16.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd498c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd44156e80 con 0x7efd441544b0 2024-03-23T15:54:16.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.601+0000 7efd42ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd440c6000 0x7efd440c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:16.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.602+0000 7efd437fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd441544b0 0x7efd440cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:16.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.602+0000 7efd427fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd440caa00 0x7efd4414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:16.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.602+0000 7efd42ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd440c6000 0x7efd440c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:56288/0 (socket says 172.21.15.129:56288) 2024-03-23T15:54:16.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.602+0000 7efd42ffd640 1 -- 172.21.15.129:0/3712364890 learned_addr learned my addr 172.21.15.129:0/3712364890 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:16.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.606+0000 7efd427fc640 1 -- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd441544b0 msgr2=0x7efd440cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:16.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.606+0000 7efd427fc640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd441544b0 0x7efd440cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:16.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.606+0000 7efd427fc640 1 -- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd440c6000 msgr2=0x7efd440c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:16.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.606+0000 7efd427fc640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd440c6000 0x7efd440c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:16.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.606+0000 7efd427fc640 1 -- 172.21.15.129:0/3712364890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd34009400 con 0x7efd440caa00 2024-03-23T15:54:16.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.608+0000 7efd427fc640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd440caa00 0x7efd4414d880 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efd3800c400 tx=0x7efd3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:16.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.611+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38005ce0 con 0x7efd440caa00 2024-03-23T15:54:16.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.611+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd38005e40 con 0x7efd440caa00 2024-03-23T15:54:16.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.611+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd380072f0 con 0x7efd440caa00 2024-03-23T15:54:16.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.611+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd440c9ec0 con 0x7efd440caa00 2024-03-23T15:54:16.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.612+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd44157400 con 0x7efd440caa00 2024-03-23T15:54:16.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.613+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd44075740 con 0x7efd440caa00 2024-03-23T15:54:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.867+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efd38007450 con 0x7efd440caa00 2024-03-23T15:54:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.868+0000 7efd2bfff640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd3006b860 0x7efd3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.868+0000 7efd42ffd640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd3006b860 0x7efd3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.869+0000 7efd42ffd640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd3006b860 0x7efd3006dd20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efd3402ba30 tx=0x7efd34008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.882+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efd38007e20 con 0x7efd440caa00 2024-03-23T15:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:16.882+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efd38054610 con 0x7efd440caa00 2024-03-23T15:54:17.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.110+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd44157d00 con 0x7efd440caa00 2024-03-23T15:54:17.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.169+0000 7efd2bfff640 1 -- 172.21.15.129:0/3712364890 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7efd38017070 con 0x7efd440caa00 2024-03-23T15:54:17.172 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:17.172 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:17.172 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:17.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd3006b860 msgr2=0x7efd3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd3006b860 0x7efd3006dd20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efd3402ba30 tx=0x7efd34008030 comp rx=0 tx=0).stop 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd440caa00 msgr2=0x7efd4414d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd440caa00 0x7efd4414d880 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efd3800c400 tx=0x7efd3800c8c0 comp rx=0 tx=0).stop 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 shutdown_connections 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd441544b0 0x7efd440cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd440caa00 0x7efd4414d880 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd3006b860 0x7efd3006dd20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 --2- 172.21.15.129:0/3712364890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd440c6000 0x7efd440c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 >> 172.21.15.129:0/3712364890 conn(0x7efd44061b50 msgr2=0x7efd440c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 shutdown_connections 2024-03-23T15:54:17.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:17.172+0000 7efd498c0640 1 -- 172.21.15.129:0/3712364890 wait complete. 2024-03-23T15:54:17.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:17.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:18.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.894+0000 7fb063857640 1 Processor -- start 2024-03-23T15:54:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.894+0000 7fb063857640 1 -- start start 2024-03-23T15:54:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.895+0000 7fb063857640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb05c0c7e70 0x7fb05c1621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.895+0000 7fb063857640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb05c073560 con 0x7fb05c150d00 2024-03-23T15:54:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.895+0000 7fb063857640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb05c072fc0 con 0x7fb05c072ad0 2024-03-23T15:54:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.895+0000 7fb063857640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb05c1510e0 con 0x7fb05c0c7e70 2024-03-23T15:54:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.896+0000 7fb061df4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb05c0c7e70 0x7fb05c1621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.896+0000 7fb060df2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb05c072ad0 0x7fb05c06d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:53002/0 (socket says 172.21.15.129:53002) 2024-03-23T15:54:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.896+0000 7fb060df2640 1 -- 172.21.15.129:0/2629606286 learned_addr learned my addr 172.21.15.129:0/2629606286 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.899+0000 7fb053fff640 1 -- 172.21.15.129:0/2629606286 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270544033 0 0) 0x7fb05c072fc0 con 0x7fb05c072ad0 2024-03-23T15:54:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.899+0000 7fb053fff640 1 -- 172.21.15.129:0/2629606286 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb048003520 con 0x7fb05c072ad0 2024-03-23T15:54:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.899+0000 7fb061df4640 1 -- 172.21.15.129:0/2629606286 >> v1:172.21.15.157:6789/0 conn(0x7fb05c072ad0 legacy=0x7fb05c06d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.899+0000 7fb061df4640 1 -- 172.21.15.129:0/2629606286 >> v1:172.21.15.129:6789/0 conn(0x7fb05c150d00 legacy=0x7fb05c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.899+0000 7fb061df4640 1 -- 172.21.15.129:0/2629606286 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c073280 con 0x7fb05c0c7e70 2024-03-23T15:54:18.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.903+0000 7fb061df4640 1 --2- 172.21.15.129:0/2629606286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1621a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb05400aea0 tx=0x7fb05402d050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f289986c863aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.907+0000 7fb053fff640 1 -- 172.21.15.129:0/2629606286 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05402dbf0 con 0x7fb05c0c7e70 2024-03-23T15:54:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.907+0000 7fb053fff640 1 -- 172.21.15.129:0/2629606286 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb05402dd50 con 0x7fb05c0c7e70 2024-03-23T15:54:18.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.907+0000 7fb053fff640 1 -- 172.21.15.129:0/2629606286 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054008660 con 0x7fb05c0c7e70 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 -- 172.21.15.129:0/2629606286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 msgr2=0x7fb05c1621a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 --2- 172.21.15.129:0/2629606286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1621a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb05400aea0 tx=0x7fb05402d050 comp rx=0 tx=0).stop 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 -- 172.21.15.129:0/2629606286 shutdown_connections 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 --2- 172.21.15.129:0/2629606286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1621a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 -- 172.21.15.129:0/2629606286 >> 172.21.15.129:0/2629606286 conn(0x7fb05c0c8370 msgr2=0x7fb05c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 -- 172.21.15.129:0/2629606286 shutdown_connections 2024-03-23T15:54:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.908+0000 7fb063857640 1 -- 172.21.15.129:0/2629606286 wait complete. 2024-03-23T15:54:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.909+0000 7fb063857640 1 Processor -- start 2024-03-23T15:54:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.909+0000 7fb063857640 1 -- start start 2024-03-23T15:54:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb063857640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb05c072ad0 0x7fb05c1ef750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb063857640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb063857640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb05c150d00 0x7fb05c1ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb063857640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1eca00 con 0x7fb05c0c7e70 2024-03-23T15:54:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb063857640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1ef2c0 con 0x7fb05c072ad0 2024-03-23T15:54:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb063857640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c0c9710 con 0x7fb05c150d00 2024-03-23T15:54:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.910+0000 7fb060df2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.911+0000 7fb0615f3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb05c072ad0 0x7fb05c1ef750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.911+0000 7fb061df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb05c150d00 0x7fb05c1ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.911+0000 7fb060df2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1ebf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44786/0 (socket says 172.21.15.129:44786) 2024-03-23T15:54:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.911+0000 7fb060df2640 1 -- 172.21.15.129:0/813759712 learned_addr learned my addr 172.21.15.129:0/813759712 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.915+0000 7fb061df4640 1 -- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb05c072ad0 msgr2=0x7fb05c1ef750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.915+0000 7fb061df4640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb05c072ad0 0x7fb05c1ef750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.915+0000 7fb061df4640 1 -- 172.21.15.129:0/813759712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 msgr2=0x7fb05c1ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.915+0000 7fb061df4640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:18.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.915+0000 7fb061df4640 1 -- 172.21.15.129:0/813759712 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05400ab80 con 0x7fb05c150d00 2024-03-23T15:54:18.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.918+0000 7fb061df4640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb05c150d00 0x7fb05c1ec4c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb05c062c00 tx=0x7fb054004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:18.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.921+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054040670 con 0x7fb05c150d00 2024-03-23T15:54:18.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.921+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c0c99f0 con 0x7fb05c150d00 2024-03-23T15:54:18.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.921+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c149c90 con 0x7fb05c150d00 2024-03-23T15:54:18.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:18.922+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c156fa0 con 0x7fb05c150d00 2024-03-23T15:54:19.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.127+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb05400a2e0 con 0x7fb05c150d00 2024-03-23T15:54:19.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.194+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0540409f0 con 0x7fb05c150d00 2024-03-23T15:54:19.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.195+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb054040c10 con 0x7fb05c150d00 2024-03-23T15:54:19.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.195+0000 7fb051ffb640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb04006b810 0x7fb04006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:19.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.195+0000 7fb0615f3640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb04006b810 0x7fb04006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:19.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.196+0000 7fb0615f3640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb04006b810 0x7fb04006dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb0580016e0 tx=0x7fb058009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:19.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.199+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb05407c200 con 0x7fb05c150d00 2024-03-23T15:54:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.209+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb0540784a0 con 0x7fb05c150d00 2024-03-23T15:54:19.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.472+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb05c156fa0 con 0x7fb05c150d00 2024-03-23T15:54:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.496+0000 7fb051ffb640 1 -- 172.21.15.129:0/813759712 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb05c156fa0 con 0x7fb05c150d00 2024-03-23T15:54:19.499 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:19.499 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:19.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb04006b810 msgr2=0x7fb04006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:19.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb04006b810 0x7fb04006dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb0580016e0 tx=0x7fb058009040 comp rx=0 tx=0).stop 2024-03-23T15:54:19.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb05c150d00 msgr2=0x7fb05c1ec4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb05c150d00 0x7fb05c1ec4c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb05c062c00 tx=0x7fb054004a40 comp rx=0 tx=0).stop 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 shutdown_connections 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb05c150d00 0x7fb05c1ec4c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb05c0c7e70 0x7fb05c1ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb04006b810 0x7fb04006dcd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 --2- 172.21.15.129:0/813759712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb05c072ad0 0x7fb05c1ef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 >> 172.21.15.129:0/813759712 conn(0x7fb05c0c8370 msgr2=0x7fb05c1e1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 shutdown_connections 2024-03-23T15:54:19.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:19.498+0000 7fb063857640 1 -- 172.21.15.129:0/813759712 wait complete. 2024-03-23T15:54:20.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:20.119 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:21.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:21.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.222+0000 7fccb1442640 1 Processor -- start 2024-03-23T15:54:21.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.222+0000 7fccb1442640 1 -- start start 2024-03-23T15:54:21.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccb1442640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fccac15a2e0 0x7fccac06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:21.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccb1442640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fccac06c0b0 0x7fccac0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:21.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccb1442640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccac0cbcb0 con 0x7fccac1544b0 2024-03-23T15:54:21.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccb1442640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fccac0c5be0 con 0x7fccac15a2e0 2024-03-23T15:54:21.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccb1442640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fccac15b280 con 0x7fccac06c0b0 2024-03-23T15:54:21.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccaa7fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fccac15a2e0 0x7fccac06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:21.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.224+0000 7fccab7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fccac06c0b0 0x7fccac0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:21.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.225+0000 7fccab7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fccac06c0b0 0x7fccac0c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43868/0 (socket says 172.21.15.129:43868) 2024-03-23T15:54:21.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.225+0000 7fccab7fe640 1 -- 172.21.15.129:0/2619695063 learned_addr learned my addr 172.21.15.129:0/2619695063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:21.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.227+0000 7fccab7fe640 1 -- 172.21.15.129:0/2619695063 >> v2:172.21.15.129:3300/0 conn(0x7fccac15a2e0 msgr2=0x7fccac06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:21.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.227+0000 7fccab7fe640 1 --2- 172.21.15.129:0/2619695063 >> v2:172.21.15.129:3300/0 conn(0x7fccac15a2e0 0x7fccac06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.227+0000 7fccab7fe640 1 -- 172.21.15.129:0/2619695063 >> v1:172.21.15.129:6789/0 conn(0x7fccac1544b0 legacy=0x7fccac0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:21.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.227+0000 7fccab7fe640 1 -- 172.21.15.129:0/2619695063 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccac15a9e0 con 0x7fccac06c0b0 2024-03-23T15:54:21.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.229+0000 7fccab7fe640 1 --2- 172.21.15.129:0/2619695063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac06c0b0 0x7fccac0c6000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fccac0b3b50 tx=0x7fcca402c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa532dc6cc7a8ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.234+0000 7fcca9ffb640 1 -- 172.21.15.129:0/2619695063 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca402cbb0 con 0x7fccac06c0b0 2024-03-23T15:54:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.234+0000 7fcca9ffb640 1 -- 172.21.15.129:0/2619695063 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca402cd10 con 0x7fccac06c0b0 2024-03-23T15:54:21.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.234+0000 7fcca9ffb640 1 -- 172.21.15.129:0/2619695063 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4034610 con 0x7fccac06c0b0 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 -- 172.21.15.129:0/2619695063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac06c0b0 msgr2=0x7fccac0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 --2- 172.21.15.129:0/2619695063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac06c0b0 0x7fccac0c6000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fccac0b3b50 tx=0x7fcca402c010 comp rx=0 tx=0).stop 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 -- 172.21.15.129:0/2619695063 shutdown_connections 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 --2- 172.21.15.129:0/2619695063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac06c0b0 0x7fccac0c6000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 --2- 172.21.15.129:0/2619695063 >> v2:172.21.15.129:3300/0 conn(0x7fccac15a2e0 0x7fccac06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 -- 172.21.15.129:0/2619695063 >> 172.21.15.129:0/2619695063 conn(0x7fccac061b50 msgr2=0x7fccac061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:21.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 -- 172.21.15.129:0/2619695063 shutdown_connections 2024-03-23T15:54:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.235+0000 7fccb1442640 1 -- 172.21.15.129:0/2619695063 wait complete. 2024-03-23T15:54:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.236+0000 7fccb1442640 1 Processor -- start 2024-03-23T15:54:21.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.236+0000 7fccb1442640 1 -- start start 2024-03-23T15:54:21.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccb1442640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccac06c0b0 0x7fccac0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:21.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccb1442640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 0x7fccac1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccb1442640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fccac15a2e0 0x7fccac0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccb1442640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fccac1f0840 con 0x7fccac06c0b0 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccb1442640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fccac1f4860 con 0x7fccac1544b0 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccb1442640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fccac1f4b40 con 0x7fccac15a2e0 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccaa7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 0x7fccac1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccaaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccac06c0b0 0x7fccac0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccaa7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 0x7fccac1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43872/0 (socket says 172.21.15.129:43872) 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccaa7fc640 1 -- 172.21.15.129:0/3858540719 learned_addr learned my addr 172.21.15.129:0/3858540719 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:21.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.238+0000 7fccab7fe640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fccac15a2e0 0x7fccac0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:21.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.242+0000 7fccaa7fc640 1 -- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fccac15a2e0 msgr2=0x7fccac0b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:21.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.242+0000 7fccaa7fc640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fccac15a2e0 0x7fccac0b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.242+0000 7fccaa7fc640 1 -- 172.21.15.129:0/3858540719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccac06c0b0 msgr2=0x7fccac0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.242+0000 7fccaa7fc640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccac06c0b0 0x7fccac0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.242+0000 7fccaa7fc640 1 -- 172.21.15.129:0/3858540719 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca4009b30 con 0x7fccac1544b0 2024-03-23T15:54:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.245+0000 7fccaa7fc640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 0x7fccac1f3e40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcca000e870 tx=0x7fcca000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:21.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.250+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca00094f0 con 0x7fccac1544b0 2024-03-23T15:54:21.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.250+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca000c940 con 0x7fccac1544b0 2024-03-23T15:54:21.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.250+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccac1f0b00 con 0x7fccac1544b0 2024-03-23T15:54:21.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.250+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca000ccf0 con 0x7fccac1544b0 2024-03-23T15:54:21.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.250+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccac1f10c0 con 0x7fccac1544b0 2024-03-23T15:54:21.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.251+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccac159280 con 0x7fccac1544b0 2024-03-23T15:54:21.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.528+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcca0013470 con 0x7fccac1544b0 2024-03-23T15:54:21.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.528+0000 7fcc937fe640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc9806b890 0x7fcc9806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:21.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.528+0000 7fccaaffd640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc9806b890 0x7fcc9806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:21.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.529+0000 7fccaaffd640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc9806b890 0x7fcc9806dd50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c001780 tx=0x7fcc9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:21.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.534+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcca0014e40 con 0x7fccac1544b0 2024-03-23T15:54:21.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.552+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcca0055460 con 0x7fccac1544b0 2024-03-23T15:54:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.778+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccac075740 con 0x7fccac1544b0 2024-03-23T15:54:21.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.793+0000 7fcc937fe640 1 -- 172.21.15.129:0/3858540719 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcca0021090 con 0x7fccac1544b0 2024-03-23T15:54:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:21.796 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:21.796 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc9806b890 msgr2=0x7fcc9806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc9806b890 0x7fcc9806dd50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c001780 tx=0x7fcc9c009040 comp rx=0 tx=0).stop 2024-03-23T15:54:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 msgr2=0x7fccac1f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 0x7fccac1f3e40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcca000e870 tx=0x7fcca000ed30 comp rx=0 tx=0).stop 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 shutdown_connections 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fccac15a2e0 0x7fccac0b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc9806b890 0x7fcc9806dd50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fccac1544b0 0x7fccac1f3e40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 --2- 172.21.15.129:0/3858540719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccac06c0b0 0x7fccac0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 >> 172.21.15.129:0/3858540719 conn(0x7fccac061b50 msgr2=0x7fccac1534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 shutdown_connections 2024-03-23T15:54:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:21.795+0000 7fccb1442640 1 -- 172.21.15.129:0/3858540719 wait complete. 2024-03-23T15:54:22.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:22.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:23.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:23.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.514+0000 7f7445793640 1 Processor -- start 2024-03-23T15:54:23.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.514+0000 7f7445793640 1 -- start start 2024-03-23T15:54:23.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.515+0000 7f7445793640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f744015a2e0 0x7f744006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:23.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.515+0000 7f7445793640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74400cbcb0 con 0x7f744006c0b0 2024-03-23T15:54:23.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.515+0000 7f7445793640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744015b280 con 0x7f74401544b0 2024-03-23T15:54:23.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.515+0000 7f7445793640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f74400c5be0 con 0x7f744015a2e0 2024-03-23T15:54:23.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.516+0000 7f7437fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f744015a2e0 0x7f744006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:23.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.516+0000 7f743effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f74401544b0 0x7f74400caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:57920/0 (socket says 172.21.15.129:57920) 2024-03-23T15:54:23.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.516+0000 7f743effd640 1 -- 172.21.15.129:0/4066719880 learned_addr learned my addr 172.21.15.129:0/4066719880 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:23.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.519+0000 7f743e7fc640 1 -- 172.21.15.129:0/4066719880 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819811418 0 0) 0x7f744015b280 con 0x7f74401544b0 2024-03-23T15:54:23.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.519+0000 7f743e7fc640 1 -- 172.21.15.129:0/4066719880 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f742c003520 con 0x7f74401544b0 2024-03-23T15:54:23.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.520+0000 7f7437fff640 1 -- 172.21.15.129:0/4066719880 >> v1:172.21.15.157:6789/0 conn(0x7f74401544b0 legacy=0x7f74400caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:23.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.520+0000 7f7437fff640 1 -- 172.21.15.129:0/4066719880 >> v1:172.21.15.129:6789/0 conn(0x7f744006c0b0 legacy=0x7f74400c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:23.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.520+0000 7f7437fff640 1 -- 172.21.15.129:0/4066719880 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744015a9e0 con 0x7f744015a2e0 2024-03-23T15:54:23.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.523+0000 7f7437fff640 1 --2- 172.21.15.129:0/4066719880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f744015a2e0 0x7f744006bb70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f74280097a0 tx=0x7f742802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa7010f3bd1af83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:23.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f743e7fc640 1 -- 172.21.15.129:0/4066719880 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428030610 con 0x7f744015a2e0 2024-03-23T15:54:23.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f743e7fc640 1 -- 172.21.15.129:0/4066719880 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7428030770 con 0x7f744015a2e0 2024-03-23T15:54:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f743e7fc640 1 -- 172.21.15.129:0/4066719880 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428030b00 con 0x7f744015a2e0 2024-03-23T15:54:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 -- 172.21.15.129:0/4066719880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f744015a2e0 msgr2=0x7f744006bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 --2- 172.21.15.129:0/4066719880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f744015a2e0 0x7f744006bb70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f74280097a0 tx=0x7f742802b960 comp rx=0 tx=0).stop 2024-03-23T15:54:23.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 -- 172.21.15.129:0/4066719880 shutdown_connections 2024-03-23T15:54:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 --2- 172.21.15.129:0/4066719880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f744015a2e0 0x7f744006bb70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 -- 172.21.15.129:0/4066719880 >> 172.21.15.129:0/4066719880 conn(0x7f7440061b50 msgr2=0x7f7440061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 -- 172.21.15.129:0/4066719880 shutdown_connections 2024-03-23T15:54:23.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.528+0000 7f7445793640 1 -- 172.21.15.129:0/4066719880 wait complete. 2024-03-23T15:54:23.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.530+0000 7f7445793640 1 Processor -- start 2024-03-23T15:54:23.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.530+0000 7f7445793640 1 -- start start 2024-03-23T15:54:23.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7445793640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f744006c0b0 0x7f7440152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:23.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7445793640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74401544b0 0x7f7440154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:23.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7445793640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f744015a2e0 0x7f74400ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:23.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7445793640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7440146140 con 0x7f74401544b0 2024-03-23T15:54:23.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7445793640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7440156640 con 0x7f744006c0b0 2024-03-23T15:54:23.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7445793640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7440156c60 con 0x7f744015a2e0 2024-03-23T15:54:23.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7437fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74401544b0 0x7f7440154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f743effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f744006c0b0 0x7f7440152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f743f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f744015a2e0 0x7f74400ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7437fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74401544b0 0x7f7440154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43672/0 (socket says 172.21.15.129:43672) 2024-03-23T15:54:23.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.531+0000 7f7437fff640 1 -- 172.21.15.129:0/2393717794 learned_addr learned my addr 172.21.15.129:0/2393717794 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.535+0000 7f743effd640 1 -- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f744015a2e0 msgr2=0x7f74400ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.535+0000 7f743effd640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f744015a2e0 0x7f74400ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.535+0000 7f743effd640 1 -- 172.21.15.129:0/2393717794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74401544b0 msgr2=0x7f7440154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.535+0000 7f743effd640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74401544b0 0x7f7440154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:23.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.535+0000 7f743effd640 1 -- 172.21.15.129:0/2393717794 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7428009400 con 0x7f744006c0b0 2024-03-23T15:54:23.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.538+0000 7f743effd640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f744006c0b0 0x7f7440152ae0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f743000c400 tx=0x7f743000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.542+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7430005ce0 con 0x7f744006c0b0 2024-03-23T15:54:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.542+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7430005e40 con 0x7f744006c0b0 2024-03-23T15:54:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.542+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74300072f0 con 0x7f744006c0b0 2024-03-23T15:54:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.542+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7440155a30 con 0x7f744006c0b0 2024-03-23T15:54:23.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.542+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7440145a80 con 0x7f744006c0b0 2024-03-23T15:54:23.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.543+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7440146280 con 0x7f744006c0b0 2024-03-23T15:54:23.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.802+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7430007450 con 0x7f744006c0b0 2024-03-23T15:54:23.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.802+0000 7f74377fe640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f742006b810 0x7f742006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:23.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.802+0000 7f7437fff640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f742006b810 0x7f742006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:23.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.803+0000 7f7437fff640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f742006b810 0x7f742006dcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7440148be0 tx=0x7f742803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:23.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.805+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7430007e20 con 0x7f744006c0b0 2024-03-23T15:54:23.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:23.817+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7430054420 con 0x7f744006c0b0 2024-03-23T15:54:24.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.038+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7440146280 con 0x7f744006c0b0 2024-03-23T15:54:24.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.053+0000 7f74377fe640 1 -- 172.21.15.129:0/2393717794 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f7440146280 con 0x7f744006c0b0 2024-03-23T15:54:24.056 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:24.056 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f742006b810 msgr2=0x7f742006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f742006b810 0x7f742006dcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7440148be0 tx=0x7f742803b040 comp rx=0 tx=0).stop 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f744006c0b0 msgr2=0x7f7440152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f744006c0b0 0x7f7440152ae0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f743000c400 tx=0x7f743000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 shutdown_connections 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f744015a2e0 0x7f74400ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74401544b0 0x7f7440154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f742006b810 0x7f742006dcd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 --2- 172.21.15.129:0/2393717794 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f744006c0b0 0x7f7440152ae0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 >> 172.21.15.129:0/2393717794 conn(0x7f7440061b50 msgr2=0x7f74400b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 shutdown_connections 2024-03-23T15:54:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:24.055+0000 7f7445793640 1 -- 172.21.15.129:0/2393717794 wait complete. 2024-03-23T15:54:24.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:24.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:25.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:25.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.774+0000 7fdcf8793640 1 Processor -- start 2024-03-23T15:54:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.774+0000 7fdcf8793640 1 -- start start 2024-03-23T15:54:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.775+0000 7fdcf8793640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdcf006c410 0x7fdcf015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.775+0000 7fdcf8793640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf015de40 0x7fdcf015e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.775+0000 7fdcf8793640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf006c830 con 0x7fdcf006c030 2024-03-23T15:54:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.775+0000 7fdcf8793640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf00cc370 con 0x7fdcf006c410 2024-03-23T15:54:25.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.775+0000 7fdcf8793640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf006cae0 con 0x7fdcf015de40 2024-03-23T15:54:25.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.776+0000 7fdcf5d2e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdcf006c410 0x7fdcf015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:25.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.776+0000 7fdcf652f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fdcf006c030 0x7fdcf00cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:57926/0 (socket says 172.21.15.129:57926) 2024-03-23T15:54:25.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.776+0000 7fdcf652f640 1 -- 172.21.15.129:0/1581871727 learned_addr learned my addr 172.21.15.129:0/1581871727 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:25.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.776+0000 7fdcf6d30640 1 --2- 172.21.15.129:0/1581871727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf015de40 0x7fdcf015e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.780+0000 7fdcf6d30640 1 -- 172.21.15.129:0/1581871727 >> v2:172.21.15.129:3300/0 conn(0x7fdcf006c410 msgr2=0x7fdcf015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.780+0000 7fdcf6d30640 1 --2- 172.21.15.129:0/1581871727 >> v2:172.21.15.129:3300/0 conn(0x7fdcf006c410 0x7fdcf015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.780+0000 7fdcf6d30640 1 -- 172.21.15.129:0/1581871727 >> v1:172.21.15.157:6789/0 conn(0x7fdcf006c030 legacy=0x7fdcf00cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.780+0000 7fdcf6d30640 1 -- 172.21.15.129:0/1581871727 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf00cabc0 con 0x7fdcf015de40 2024-03-23T15:54:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.780+0000 7fdcf5d2e640 1 --2- 172.21.15.129:0/1581871727 >> v2:172.21.15.129:3300/0 conn(0x7fdcf006c410 0x7fdcf015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:54:25.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.782+0000 7fdcf6d30640 1 --2- 172.21.15.129:0/1581871727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf015de40 0x7fdcf015e2b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009d90 tx=0x7fdce802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80c4229916689ada server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:25.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.786+0000 7fdcf552d640 1 -- 172.21.15.129:0/1581871727 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce802cc50 con 0x7fdcf015de40 2024-03-23T15:54:25.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.786+0000 7fdcf552d640 1 -- 172.21.15.129:0/1581871727 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce802cdb0 con 0x7fdcf015de40 2024-03-23T15:54:25.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.786+0000 7fdcf552d640 1 -- 172.21.15.129:0/1581871727 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8034660 con 0x7fdcf015de40 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 -- 172.21.15.129:0/1581871727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf015de40 msgr2=0x7fdcf015e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 --2- 172.21.15.129:0/1581871727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf015de40 0x7fdcf015e2b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009d90 tx=0x7fdce802bea0 comp rx=0 tx=0).stop 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 -- 172.21.15.129:0/1581871727 shutdown_connections 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 --2- 172.21.15.129:0/1581871727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf015de40 0x7fdcf015e2b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 --2- 172.21.15.129:0/1581871727 >> v2:172.21.15.129:3300/0 conn(0x7fdcf006c410 0x7fdcf015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 -- 172.21.15.129:0/1581871727 >> 172.21.15.129:0/1581871727 conn(0x7fdcf00cfb20 msgr2=0x7fdcf00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 -- 172.21.15.129:0/1581871727 shutdown_connections 2024-03-23T15:54:25.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.787+0000 7fdcf8793640 1 -- 172.21.15.129:0/1581871727 wait complete. 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.788+0000 7fdcf8793640 1 Processor -- start 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.788+0000 7fdcf8793640 1 -- start start 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf8793640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf006c030 0x7fdcf0155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf8793640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf006c410 0x7fdcf015d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf8793640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 0x7fdcf014e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf8793640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf00b2740 con 0x7fdcf015de40 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf8793640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf01553a0 con 0x7fdcf006c030 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf8793640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf00b4190 con 0x7fdcf006c410 2024-03-23T15:54:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf6d30640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 0x7fdcf014e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf5d2e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf006c410 0x7fdcf015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.789+0000 7fdcf652f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf006c030 0x7fdcf0155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.790+0000 7fdcf6d30640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 0x7fdcf014e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43682/0 (socket says 172.21.15.129:43682) 2024-03-23T15:54:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.790+0000 7fdcf6d30640 1 -- 172.21.15.129:0/3043879218 learned_addr learned my addr 172.21.15.129:0/3043879218 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.793+0000 7fdcf6d30640 1 -- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf006c410 msgr2=0x7fdcf015d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:25.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.793+0000 7fdcf6d30640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf006c410 0x7fdcf015d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.793+0000 7fdcf6d30640 1 -- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf006c030 msgr2=0x7fdcf0155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.793+0000 7fdcf6d30640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf006c030 0x7fdcf0155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:25.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.793+0000 7fdcf6d30640 1 -- 172.21.15.129:0/3043879218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce0009400 con 0x7fdcf015de40 2024-03-23T15:54:25.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.796+0000 7fdcf6d30640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 0x7fdcf014e020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009eb0 tx=0x7fdce8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.799+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce803e610 con 0x7fdcf015de40 2024-03-23T15:54:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.799+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce8009a70 con 0x7fdcf015de40 2024-03-23T15:54:25.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.799+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf00b2950 con 0x7fdcf015de40 2024-03-23T15:54:25.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:25.801+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf0075740 con 0x7fdcf015de40 2024-03-23T15:54:26.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.066+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce803e770 con 0x7fdcf015de40 2024-03-23T15:54:26.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.066+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce803eae0 con 0x7fdcf015de40 2024-03-23T15:54:26.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.067+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdce803b050 con 0x7fdcf015de40 2024-03-23T15:54:26.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.067+0000 7fdce6ffd640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdcd406b810 0x7fdcd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:26.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.067+0000 7fdcf652f640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdcd406b810 0x7fdcd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:26.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.068+0000 7fdcf652f640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdcd406b810 0x7fdcd406dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007fd0 tx=0x7fdcec007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:26.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.082+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdce807b270 con 0x7fdcf015de40 2024-03-23T15:54:26.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.082+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdce8077510 con 0x7fdcf015de40 2024-03-23T15:54:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.328+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcf00d0a30 con 0x7fdcf015de40 2024-03-23T15:54:26.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.341+0000 7fdce6ffd640 1 -- 172.21.15.129:0/3043879218 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fdce804e020 con 0x7fdcf015de40 2024-03-23T15:54:26.344 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:26.344 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:26.344 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:26.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdcd406b810 msgr2=0x7fdcd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:26.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdcd406b810 0x7fdcd406dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007fd0 tx=0x7fdcec007e80 comp rx=0 tx=0).stop 2024-03-23T15:54:26.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 msgr2=0x7fdcf014e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 0x7fdcf014e020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009eb0 tx=0x7fdce8004a40 comp rx=0 tx=0).stop 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 shutdown_connections 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcf015de40 0x7fdcf014e020 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdcf006c410 0x7fdcf015d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdcd406b810 0x7fdcd406dcd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 --2- 172.21.15.129:0/3043879218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcf006c030 0x7fdcf0155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 >> 172.21.15.129:0/3043879218 conn(0x7fdcf00cfb20 msgr2=0x7fdcf015c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 shutdown_connections 2024-03-23T15:54:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:26.343+0000 7fdcf8793640 1 -- 172.21.15.129:0/3043879218 wait complete. 2024-03-23T15:54:26.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:26.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:27.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.064+0000 7f3ee382c640 1 Processor -- start 2024-03-23T15:54:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.064+0000 7f3ee382c640 1 -- start start 2024-03-23T15:54:28.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.065+0000 7f3ee382c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3edc0ccd60 0x7f3edc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:28.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.065+0000 7f3ee382c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3edc0ceb00 0x7f3edc06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:28.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.065+0000 7f3ee382c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3edc15b290 con 0x7f3edc0cef40 2024-03-23T15:54:28.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.065+0000 7f3ee382c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3edc0ce800 con 0x7f3edc0ceb00 2024-03-23T15:54:28.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.065+0000 7f3ee382c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3edc154a70 con 0x7f3edc0ccd60 2024-03-23T15:54:28.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.066+0000 7f3ee15c8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3edc0ccd60 0x7f3edc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:28.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.066+0000 7f3ee0dc7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3edc0ceb00 0x7f3edc06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:28.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.066+0000 7f3ee15c8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3edc0ccd60 0x7f3edc15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43900/0 (socket says 172.21.15.129:43900) 2024-03-23T15:54:28.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.066+0000 7f3ee15c8640 1 -- 172.21.15.129:0/3168362703 learned_addr learned my addr 172.21.15.129:0/3168362703 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:28.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.068+0000 7f3ee15c8640 1 -- 172.21.15.129:0/3168362703 >> v2:172.21.15.129:3300/0 conn(0x7f3edc0ceb00 msgr2=0x7f3edc06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:28.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.068+0000 7f3ee15c8640 1 --2- 172.21.15.129:0/3168362703 >> v2:172.21.15.129:3300/0 conn(0x7f3edc0ceb00 0x7f3edc06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.068+0000 7f3ee15c8640 1 -- 172.21.15.129:0/3168362703 >> v1:172.21.15.129:6789/0 conn(0x7f3edc0cef40 legacy=0x7f3edc15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:28.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.068+0000 7f3ee15c8640 1 -- 172.21.15.129:0/3168362703 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc0cd950 con 0x7f3edc0ccd60 2024-03-23T15:54:28.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.070+0000 7f3ee15c8640 1 --2- 172.21.15.129:0/3168362703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ccd60 0x7f3edc15ad50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3edc08cf40 tx=0x7f3ed802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abceb3886fa17c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:28.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ed3fff640 1 -- 172.21.15.129:0/3168362703 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed802f850 con 0x7f3edc0ccd60 2024-03-23T15:54:28.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ed3fff640 1 -- 172.21.15.129:0/3168362703 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed802f9b0 con 0x7f3edc0ccd60 2024-03-23T15:54:28.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ed3fff640 1 -- 172.21.15.129:0/3168362703 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed802fd40 con 0x7f3edc0ccd60 2024-03-23T15:54:28.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 -- 172.21.15.129:0/3168362703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ccd60 msgr2=0x7f3edc15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:28.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 --2- 172.21.15.129:0/3168362703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ccd60 0x7f3edc15ad50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3edc08cf40 tx=0x7f3ed802ba30 comp rx=0 tx=0).stop 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 -- 172.21.15.129:0/3168362703 shutdown_connections 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 --2- 172.21.15.129:0/3168362703 >> v2:172.21.15.129:3300/0 conn(0x7f3edc0ceb00 0x7f3edc06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 --2- 172.21.15.129:0/3168362703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ccd60 0x7f3edc15ad50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 -- 172.21.15.129:0/3168362703 >> 172.21.15.129:0/3168362703 conn(0x7f3edc0b1840 msgr2=0x7f3edc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 -- 172.21.15.129:0/3168362703 shutdown_connections 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.075+0000 7f3ee382c640 1 -- 172.21.15.129:0/3168362703 wait complete. 2024-03-23T15:54:28.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 Processor -- start 2024-03-23T15:54:28.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 -- start start 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3edc0ccd60 0x7f3edc159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 0x7f3edc159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3edc0cef40 0x7f3edc15cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc1592d0 con 0x7f3edc0ccd60 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc158fd0 con 0x7f3edc0ceb00 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.077+0000 7f3ee382c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3edc1e4390 con 0x7f3edc0cef40 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.078+0000 7f3ee0dc7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 0x7f3edc159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.078+0000 7f3ee15c8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3edc0ccd60 0x7f3edc159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.078+0000 7f3ee0dc7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 0x7f3edc159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43912/0 (socket says 172.21.15.129:43912) 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.078+0000 7f3ee0dc7640 1 -- 172.21.15.129:0/4217180076 learned_addr learned my addr 172.21.15.129:0/4217180076 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:28.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.078+0000 7f3ee1dc9640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3edc0cef40 0x7f3edc15cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:28.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.083+0000 7f3ee0dc7640 1 -- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3edc0cef40 msgr2=0x7f3edc15cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:28.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.083+0000 7f3ee0dc7640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3edc0cef40 0x7f3edc15cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.083+0000 7f3ee0dc7640 1 -- 172.21.15.129:0/4217180076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3edc0ccd60 msgr2=0x7f3edc159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:28.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.083+0000 7f3ee0dc7640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3edc0ccd60 0x7f3edc159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.083+0000 7f3ee0dc7640 1 -- 172.21.15.129:0/4217180076 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed80094d0 con 0x7f3edc0ceb00 2024-03-23T15:54:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.086+0000 7f3ee0dc7640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 0x7f3edc159ca0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00d800 tx=0x7f3ecc00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:28.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.090+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00b6c0 con 0x7f3edc0ceb00 2024-03-23T15:54:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.090+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ecc00b820 con 0x7f3edc0ceb00 2024-03-23T15:54:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.090+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc1e4670 con 0x7f3edc0ceb00 2024-03-23T15:54:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.090+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00bbb0 con 0x7f3edc0ceb00 2024-03-23T15:54:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.091+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1e4050 con 0x7f3edc0ceb00 2024-03-23T15:54:28.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.092+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc04f530 con 0x7f3edc0ceb00 2024-03-23T15:54:28.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.123+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ecc00bd10 con 0x7f3edc0ceb00 2024-03-23T15:54:28.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.123+0000 7f3ed1ffb640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ec006b860 0x7f3ec006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:28.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.123+0000 7f3ee15c8640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ec006b860 0x7f3ec006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:28.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.124+0000 7f3ee15c8640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ec006b860 0x7f3ec006dd20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8036000 tx=0x7f3ed8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:28.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.126+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3ecc059970 con 0x7f3edc0ceb00 2024-03-23T15:54:28.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.374+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ecc055c10 con 0x7f3edc0ceb00 2024-03-23T15:54:28.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.628+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3edc06db50 con 0x7f3edc0ceb00 2024-03-23T15:54:28.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.744+0000 7f3ed1ffb640 1 -- 172.21.15.129:0/4217180076 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3ecc0592d0 con 0x7f3edc0ceb00 2024-03-23T15:54:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:28.748 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:28.748 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:28.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ec006b860 msgr2=0x7f3ec006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:28.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ec006b860 0x7f3ec006dd20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8036000 tx=0x7f3ed8008030 comp rx=0 tx=0).stop 2024-03-23T15:54:28.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 msgr2=0x7f3edc159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:28.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 0x7f3edc159ca0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00d800 tx=0x7f3ecc00dcc0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 shutdown_connections 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3edc0cef40 0x7f3edc15cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ec006b860 0x7f3ec006dd20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3edc0ceb00 0x7f3edc159ca0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 --2- 172.21.15.129:0/4217180076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3edc0ccd60 0x7f3edc159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 >> 172.21.15.129:0/4217180076 conn(0x7f3edc0b1840 msgr2=0x7f3edc158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 shutdown_connections 2024-03-23T15:54:28.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:28.747+0000 7f3ee382c640 1 -- 172.21.15.129:0/4217180076 wait complete. 2024-03-23T15:54:29.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:29.370 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:30.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:30.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.476+0000 7fd59bcb5640 1 Processor -- start 2024-03-23T15:54:30.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.476+0000 7fd59bcb5640 1 -- start start 2024-03-23T15:54:30.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd59bcb5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd59406c410 0x7fd59415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:30.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd59bcb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:30.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd59bcb5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59406c830 con 0x7fd59406c030 2024-03-23T15:54:30.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd59bcb5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd5940cc370 con 0x7fd59406c410 2024-03-23T15:54:30.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd59bcb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd59406cae0 con 0x7fd59415de40 2024-03-23T15:54:30.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd599250640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd59406c410 0x7fd59415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:30.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.477+0000 7fd59a252640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:30.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.478+0000 7fd599a51640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd59406c030 0x7fd5940cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:57930/0 (socket says 172.21.15.129:57930) 2024-03-23T15:54:30.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.478+0000 7fd599a51640 1 -- 172.21.15.129:0/2864065613 learned_addr learned my addr 172.21.15.129:0/2864065613 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:30.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.480+0000 7fd599250640 1 -- 172.21.15.129:0/2864065613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 msgr2=0x7fd59415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:30.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.480+0000 7fd599250640 1 --2- 172.21.15.129:0/2864065613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59415e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:30.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.480+0000 7fd599250640 1 -- 172.21.15.129:0/2864065613 >> v1:172.21.15.157:6789/0 conn(0x7fd59406c030 legacy=0x7fd5940cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:30.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.480+0000 7fd599250640 1 -- 172.21.15.129:0/2864065613 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5940cabc0 con 0x7fd59406c410 2024-03-23T15:54:30.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.482+0000 7fd599250640 1 --2- 172.21.15.129:0/2864065613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c410 0x7fd59415d900 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd5840097a0 tx=0x7fd58402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7259cfb103790ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:30.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd598a4f640 1 -- 172.21.15.129:0/2864065613 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584031830 con 0x7fd59406c410 2024-03-23T15:54:30.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd598a4f640 1 -- 172.21.15.129:0/2864065613 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd584031990 con 0x7fd59406c410 2024-03-23T15:54:30.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd598a4f640 1 -- 172.21.15.129:0/2864065613 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584031d20 con 0x7fd59406c410 2024-03-23T15:54:30.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 -- 172.21.15.129:0/2864065613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c410 msgr2=0x7fd59415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:30.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/2864065613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c410 0x7fd59415d900 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd5840097a0 tx=0x7fd58402b8d0 comp rx=0 tx=0).stop 2024-03-23T15:54:30.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 -- 172.21.15.129:0/2864065613 shutdown_connections 2024-03-23T15:54:30.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/2864065613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59415e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/2864065613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c410 0x7fd59415d900 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 -- 172.21.15.129:0/2864065613 >> 172.21.15.129:0/2864065613 conn(0x7fd5940cfb20 msgr2=0x7fd5940cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 -- 172.21.15.129:0/2864065613 shutdown_connections 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.487+0000 7fd59bcb5640 1 -- 172.21.15.129:0/2864065613 wait complete. 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 Processor -- start 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 -- start start 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c030 0x7fd5940b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd59406c410 0x7fd5941f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd59414ed20 con 0x7fd59406c030 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd5941efde0 con 0x7fd59406c410 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.489+0000 7fd59bcb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd594154d00 con 0x7fd59415de40 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.490+0000 7fd59a252640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:30.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.490+0000 7fd599250640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd59406c410 0x7fd5941f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.491+0000 7fd59a252640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59414df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:55090/0 (socket says 172.21.15.129:55090) 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.491+0000 7fd59a252640 1 -- 172.21.15.129:0/68213498 learned_addr learned my addr 172.21.15.129:0/68213498 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.491+0000 7fd599a51640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c030 0x7fd5940b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.495+0000 7fd599250640 1 -- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 msgr2=0x7fd59414df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.495+0000 7fd599250640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59414df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.495+0000 7fd599250640 1 -- 172.21.15.129:0/68213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c030 msgr2=0x7fd5940b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.495+0000 7fd599250640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c030 0x7fd5940b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:30.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.495+0000 7fd599250640 1 -- 172.21.15.129:0/68213498 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd584009400 con 0x7fd59406c410 2024-03-23T15:54:30.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.498+0000 7fd599250640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd59406c410 0x7fd5941f0510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd584033000 tx=0x7fd5840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:30.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.506+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58403a6f0 con 0x7fd59406c410 2024-03-23T15:54:30.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.506+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58403a870 con 0x7fd59406c410 2024-03-23T15:54:30.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.506+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5941f40a0 con 0x7fd59406c410 2024-03-23T15:54:30.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.506+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58403ac20 con 0x7fd59406c410 2024-03-23T15:54:30.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.506+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd594155230 con 0x7fd59406c410 2024-03-23T15:54:30.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.507+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5940c96e0 con 0x7fd59406c410 2024-03-23T15:54:30.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.533+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd58404c430 con 0x7fd59406c410 2024-03-23T15:54:30.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.533+0000 7fd58a7fc640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd57806b890 0x7fd57806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:30.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.533+0000 7fd599a51640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd57806b890 0x7fd57806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:30.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.534+0000 7fd599a51640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd57806b890 0x7fd57806dd50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd590007fd0 tx=0x7fd590007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:30.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.537+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd58407abc0 con 0x7fd59406c410 2024-03-23T15:54:30.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:30.795+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd584076e60 con 0x7fd59406c410 2024-03-23T15:54:31.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.015+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd594075740 con 0x7fd59406c410 2024-03-23T15:54:31.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.033+0000 7fd58a7fc640 1 -- 172.21.15.129:0/68213498 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd58407a520 con 0x7fd59406c410 2024-03-23T15:54:31.036 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:31.036 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:31.037 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd57806b890 msgr2=0x7fd57806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd57806b890 0x7fd57806dd50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd590007fd0 tx=0x7fd590007e80 comp rx=0 tx=0).stop 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd59406c410 msgr2=0x7fd5941f0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd59406c410 0x7fd5941f0510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd584033000 tx=0x7fd5840046c0 comp rx=0 tx=0).stop 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 shutdown_connections 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd59415de40 0x7fd59414df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd57806b890 0x7fd57806dd50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:31.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd59406c410 0x7fd5941f0510 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:31.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 --2- 172.21.15.129:0/68213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd59406c030 0x7fd5940b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:31.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 >> 172.21.15.129:0/68213498 conn(0x7fd5940cfb20 msgr2=0x7fd5940c8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:31.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 shutdown_connections 2024-03-23T15:54:31.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:31.035+0000 7fd59bcb5640 1 -- 172.21.15.129:0/68213498 wait complete. 2024-03-23T15:54:31.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:31.654 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:32.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:32.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.756+0000 7f97d20c9640 1 Processor -- start 2024-03-23T15:54:32.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.756+0000 7f97d20c9640 1 -- start start 2024-03-23T15:54:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.757+0000 7f97d20c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f97cc06f570 0x7f97cc06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.757+0000 7f97d20c9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc06d2c0 con 0x7f97cc15e040 2024-03-23T15:54:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.757+0000 7f97d20c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc15e740 con 0x7f97cc15dc60 2024-03-23T15:54:32.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.757+0000 7f97d20c9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f97cc0ccd00 con 0x7f97cc06f570 2024-03-23T15:54:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.758+0000 7f97cbfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f97cc06f570 0x7f97cc06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.758+0000 7f97cbfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f97cc06f570 0x7f97cc06cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40538/0 (socket says 172.21.15.129:40538) 2024-03-23T15:54:32.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.758+0000 7f97cbfff640 1 -- 172.21.15.129:0/3954711511 learned_addr learned my addr 172.21.15.129:0/3954711511 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.761+0000 7f97cbfff640 1 -- 172.21.15.129:0/3954711511 >> v1:172.21.15.157:6789/0 conn(0x7f97cc15dc60 legacy=0x7f97cc0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.762+0000 7f97cbfff640 1 -- 172.21.15.129:0/3954711511 >> v1:172.21.15.129:6789/0 conn(0x7f97cc15e040 legacy=0x7f97cc0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:32.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.762+0000 7f97cbfff640 1 -- 172.21.15.129:0/3954711511 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc0ccfe0 con 0x7f97cc06f570 2024-03-23T15:54:32.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.766+0000 7f97cbfff640 1 --2- 172.21.15.129:0/3954711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 0x7f97cc06cc50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009e50 tx=0x7f97c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4960ffcfda1c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:32.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.770+0000 7f97ca7fc640 1 -- 172.21.15.129:0/3954711511 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c402ca80 con 0x7f97cc06f570 2024-03-23T15:54:32.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.770+0000 7f97ca7fc640 1 -- 172.21.15.129:0/3954711511 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c402cbe0 con 0x7f97cc06f570 2024-03-23T15:54:32.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.770+0000 7f97ca7fc640 1 -- 172.21.15.129:0/3954711511 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c40343f0 con 0x7f97cc06f570 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 -- 172.21.15.129:0/3954711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 msgr2=0x7f97cc06cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 --2- 172.21.15.129:0/3954711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 0x7f97cc06cc50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009e50 tx=0x7f97c402bee0 comp rx=0 tx=0).stop 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 -- 172.21.15.129:0/3954711511 shutdown_connections 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 --2- 172.21.15.129:0/3954711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 0x7f97cc06cc50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 -- 172.21.15.129:0/3954711511 >> 172.21.15.129:0/3954711511 conn(0x7f97cc071300 msgr2=0x7f97cc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 -- 172.21.15.129:0/3954711511 shutdown_connections 2024-03-23T15:54:32.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.771+0000 7f97d20c9640 1 -- 172.21.15.129:0/3954711511 wait complete. 2024-03-23T15:54:32.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.773+0000 7f97d20c9640 1 Processor -- start 2024-03-23T15:54:32.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.773+0000 7f97d20c9640 1 -- start start 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.774+0000 7f97d20c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 0x7f97cc15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.774+0000 7f97d20c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97cc15dc60 0x7f97cc1f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.774+0000 7f97d20c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 0x7f97cc1f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.774+0000 7f97d20c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1e6c20 con 0x7f97cc06f570 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.774+0000 7f97d20c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc1e3a70 con 0x7f97cc15e040 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.774+0000 7f97d20c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97cc1f3260 con 0x7f97cc15dc60 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.775+0000 7f97caffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97cc15dc60 0x7f97cc1f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:32.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.775+0000 7f97cbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 0x7f97cc1f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:32.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.775+0000 7f97cbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 0x7f97cc1f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:33906/0 (socket says 172.21.15.129:33906) 2024-03-23T15:54:32.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.775+0000 7f97cbfff640 1 -- 172.21.15.129:0/2460808278 learned_addr learned my addr 172.21.15.129:0/2460808278 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:32.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.779+0000 7f97cbfff640 1 -- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97cc15dc60 msgr2=0x7f97cc1f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:32.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.779+0000 7f97cbfff640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97cc15dc60 0x7f97cc1f36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:32.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.779+0000 7f97cbfff640 1 -- 172.21.15.129:0/2460808278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 msgr2=0x7f97cc15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:32.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.779+0000 7f97cbfff640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 0x7f97cc15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:32.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.779+0000 7f97cbfff640 1 -- 172.21.15.129:0/2460808278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c4009b30 con 0x7f97cc15e040 2024-03-23T15:54:32.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.783+0000 7f97cbfff640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 0x7f97cc1f3c30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009e50 tx=0x7f97c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:32.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.788+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c403e450 con 0x7f97cc15e040 2024-03-23T15:54:32.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.788+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c403e5d0 con 0x7f97cc15e040 2024-03-23T15:54:32.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.788+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc1e7130 con 0x7f97cc15e040 2024-03-23T15:54:32.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.788+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c403e980 con 0x7f97cc15e040 2024-03-23T15:54:32.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.788+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97cc1e3f60 con 0x7f97cc15e040 2024-03-23T15:54:32.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.789+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97cc075740 con 0x7f97cc15e040 2024-03-23T15:54:32.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.814+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97c402cd50 con 0x7f97cc15e040 2024-03-23T15:54:32.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.814+0000 7f97b3fff640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97b806b860 0x7f97b806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:32.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.815+0000 7f97cb7fe640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97b806b860 0x7f97b806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:32.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:32.815+0000 7f97cb7fe640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97b806b860 0x7f97b806dd20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f97bc001560 tx=0x7f97bc007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:33.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.106+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f97c407c380 con 0x7f97cc15e040 2024-03-23T15:54:33.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.127+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97c4077f20 con 0x7f97cc15e040 2024-03-23T15:54:33.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.356+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97cc06d9c0 con 0x7f97cc15e040 2024-03-23T15:54:33.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.380+0000 7f97b3fff640 1 -- 172.21.15.129:0/2460808278 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f97c4043020 con 0x7f97cc15e040 2024-03-23T15:54:33.383 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:33.383 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:33.384 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:33.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97b806b860 msgr2=0x7f97b806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97b806b860 0x7f97b806dd20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f97bc001560 tx=0x7f97bc007990 comp rx=0 tx=0).stop 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 msgr2=0x7f97cc1f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 0x7f97cc1f3c30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009e50 tx=0x7f97c4004a40 comp rx=0 tx=0).stop 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 shutdown_connections 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97b806b860 0x7f97b806dd20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97cc15e040 0x7f97cc1f3c30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97cc15dc60 0x7f97cc1f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 --2- 172.21.15.129:0/2460808278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97cc06f570 0x7f97cc15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.382+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 >> 172.21.15.129:0/2460808278 conn(0x7f97cc071300 msgr2=0x7f97cc0c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.383+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 shutdown_connections 2024-03-23T15:54:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:33.383+0000 7f97d20c9640 1 -- 172.21.15.129:0/2460808278 wait complete. 2024-03-23T15:54:34.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:34.003 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:35.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.104+0000 7ff1b9d81640 1 Processor -- start 2024-03-23T15:54:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.104+0000 7ff1b9d81640 1 -- start start 2024-03-23T15:54:35.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.105+0000 7ff1b9d81640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff1b40ccd60 0x7ff1b415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:35.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.105+0000 7ff1b9d81640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b415b290 con 0x7ff1b40cef40 2024-03-23T15:54:35.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.105+0000 7ff1b9d81640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b40ce800 con 0x7ff1b40ceb00 2024-03-23T15:54:35.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.105+0000 7ff1b9d81640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b4154a70 con 0x7ff1b40ccd60 2024-03-23T15:54:35.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.106+0000 7ff1b2ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff1b40ceb00 0x7ff1b406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:48998/0 (socket says 172.21.15.129:48998) 2024-03-23T15:54:35.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.106+0000 7ff1b2ffd640 1 -- 172.21.15.129:0/3519429866 learned_addr learned my addr 172.21.15.129:0/3519429866 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:35.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.107+0000 7ff1b37fe640 1 --2- 172.21.15.129:0/3519429866 >> v2:172.21.15.129:3300/0 conn(0x7ff1b40ccd60 0x7ff1b415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:35.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.108+0000 7ff1b27fc640 1 -- 172.21.15.129:0/3519429866 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944147840 0 0) 0x7ff1b40ce800 con 0x7ff1b40ceb00 2024-03-23T15:54:35.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.108+0000 7ff1b27fc640 1 -- 172.21.15.129:0/3519429866 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff19c003520 con 0x7ff1b40ceb00 2024-03-23T15:54:35.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.111+0000 7ff1b37fe640 1 -- 172.21.15.129:0/3519429866 >> v1:172.21.15.157:6789/0 conn(0x7ff1b40ceb00 legacy=0x7ff1b406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.111+0000 7ff1b37fe640 1 -- 172.21.15.129:0/3519429866 >> v1:172.21.15.129:6789/0 conn(0x7ff1b40cef40 legacy=0x7ff1b415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.111+0000 7ff1b37fe640 1 -- 172.21.15.129:0/3519429866 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b40cd950 con 0x7ff1b40ccd60 2024-03-23T15:54:35.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.117+0000 7ff1b37fe640 1 --2- 172.21.15.129:0/3519429866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ccd60 0x7ff1b415ad50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40097a0 tx=0x7ff1a402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb7c36b0a1ff7be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:35.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.121+0000 7ff1b27fc640 1 -- 172.21.15.129:0/3519429866 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a402f810 con 0x7ff1b40ccd60 2024-03-23T15:54:35.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.121+0000 7ff1b27fc640 1 -- 172.21.15.129:0/3519429866 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a402f970 con 0x7ff1b40ccd60 2024-03-23T15:54:35.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.121+0000 7ff1b27fc640 1 -- 172.21.15.129:0/3519429866 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a402fd00 con 0x7ff1b40ccd60 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.122+0000 7ff1b9d81640 1 -- 172.21.15.129:0/3519429866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ccd60 msgr2=0x7ff1b415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.122+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/3519429866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ccd60 0x7ff1b415ad50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40097a0 tx=0x7ff1a402b9f0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.122+0000 7ff1b9d81640 1 -- 172.21.15.129:0/3519429866 shutdown_connections 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.122+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/3519429866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ccd60 0x7ff1b415ad50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.123+0000 7ff1b9d81640 1 -- 172.21.15.129:0/3519429866 >> 172.21.15.129:0/3519429866 conn(0x7ff1b40b1840 msgr2=0x7ff1b40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.123+0000 7ff1b9d81640 1 -- 172.21.15.129:0/3519429866 shutdown_connections 2024-03-23T15:54:35.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.123+0000 7ff1b9d81640 1 -- 172.21.15.129:0/3519429866 wait complete. 2024-03-23T15:54:35.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.124+0000 7ff1b9d81640 1 Processor -- start 2024-03-23T15:54:35.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.125+0000 7ff1b9d81640 1 -- start start 2024-03-23T15:54:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b9d81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b40ccd60 0x7ff1b4152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b9d81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 0x7ff1b41f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b9d81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1b40cef40 0x7ff1b41474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b9d81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b4151ac0 con 0x7ff1b40ceb00 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b9d81640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41f0e00 con 0x7ff1b40ccd60 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b9d81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b414b7b0 con 0x7ff1b40cef40 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 0x7ff1b41f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.126+0000 7ff1b37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b40ccd60 0x7ff1b4152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:35.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.127+0000 7ff1b3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1b40cef40 0x7ff1b41474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:35.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.127+0000 7ff1b2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 0x7ff1b41f08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40550/0 (socket says 172.21.15.129:40550) 2024-03-23T15:54:35.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.127+0000 7ff1b2ffd640 1 -- 172.21.15.129:0/1234667065 learned_addr learned my addr 172.21.15.129:0/1234667065 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:35.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.132+0000 7ff1b2ffd640 1 -- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1b40cef40 msgr2=0x7ff1b41474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.132+0000 7ff1b2ffd640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1b40cef40 0x7ff1b41474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.132+0000 7ff1b2ffd640 1 -- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b40ccd60 msgr2=0x7ff1b4152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.132+0000 7ff1b2ffd640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b40ccd60 0x7ff1b4152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.132+0000 7ff1b2ffd640 1 -- 172.21.15.129:0/1234667065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a4009400 con 0x7ff1b40ceb00 2024-03-23T15:54:35.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.134+0000 7ff1b2ffd640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 0x7ff1b41f08c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800c530 tx=0x7ff1a800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:35.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.139+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8005ce0 con 0x7ff1b40ceb00 2024-03-23T15:54:35.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.139+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a8005e40 con 0x7ff1b40ceb00 2024-03-23T15:54:35.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.139+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b4151500 con 0x7ff1b40ceb00 2024-03-23T15:54:35.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.139+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a80072b0 con 0x7ff1b40ceb00 2024-03-23T15:54:35.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.139+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b414ba50 con 0x7ff1b40ceb00 2024-03-23T15:54:35.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.140+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b404f530 con 0x7ff1b40ceb00 2024-03-23T15:54:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.393+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff1a8007410 con 0x7ff1b40ceb00 2024-03-23T15:54:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.394+0000 7ff19bfff640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1a006b840 0x7ff1a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.394+0000 7ff1b37fe640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1a006b840 0x7ff1a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.395+0000 7ff1b37fe640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1a006b840 0x7ff1a006dd00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4036000 tx=0x7ff1a4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:35.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.407+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff1a8058420 con 0x7ff1b40ceb00 2024-03-23T15:54:35.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.407+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff1a80581f0 con 0x7ff1b40ceb00 2024-03-23T15:54:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.635+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1b40686f0 con 0x7ff1b40ceb00 2024-03-23T15:54:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.658+0000 7ff19bfff640 1 -- 172.21.15.129:0/1234667065 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff1a8017020 con 0x7ff1b40ceb00 2024-03-23T15:54:35.661 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:35.661 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1a006b840 msgr2=0x7ff1a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1a006b840 0x7ff1a006dd00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4036000 tx=0x7ff1a4008030 comp rx=0 tx=0).stop 2024-03-23T15:54:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 msgr2=0x7ff1b41f08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 0x7ff1b41f08c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800c530 tx=0x7ff1a800c9f0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 shutdown_connections 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1b40cef40 0x7ff1b41474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1b40ceb00 0x7ff1b41f08c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff1a006b840 0x7ff1a006dd00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 --2- 172.21.15.129:0/1234667065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1b40ccd60 0x7ff1b4152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 >> 172.21.15.129:0/1234667065 conn(0x7ff1b40b1840 msgr2=0x7ff1b4158550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 shutdown_connections 2024-03-23T15:54:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:35.660+0000 7ff1b9d81640 1 -- 172.21.15.129:0/1234667065 wait complete. 2024-03-23T15:54:36.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:36.279 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:37.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:37.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.380+0000 7f343f40e640 1 Processor -- start 2024-03-23T15:54:37.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.381+0000 7f343f40e640 1 -- start start 2024-03-23T15:54:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.381+0000 7f343f40e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f343815a2e0 0x7f343806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.381+0000 7f343f40e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f343806c0b0 0x7f34380c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.381+0000 7f343f40e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34380cbcb0 con 0x7f34381544b0 2024-03-23T15:54:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.381+0000 7f343f40e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f34380c5be0 con 0x7f343815a2e0 2024-03-23T15:54:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.381+0000 7f343f40e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f343815b280 con 0x7f343806c0b0 2024-03-23T15:54:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.382+0000 7f343d9ab640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f343806c0b0 0x7f34380c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.382+0000 7f343c9a9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f343815a2e0 0x7f343806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.382+0000 7f343d9ab640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f343806c0b0 0x7f34380c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:33928/0 (socket says 172.21.15.129:33928) 2024-03-23T15:54:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.382+0000 7f343d9ab640 1 -- 172.21.15.129:0/4001353659 learned_addr learned my addr 172.21.15.129:0/4001353659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.385+0000 7f343d9ab640 1 -- 172.21.15.129:0/4001353659 >> v2:172.21.15.129:3300/0 conn(0x7f343815a2e0 msgr2=0x7f343806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.385+0000 7f343d9ab640 1 --2- 172.21.15.129:0/4001353659 >> v2:172.21.15.129:3300/0 conn(0x7f343815a2e0 0x7f343806bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.385+0000 7f343d9ab640 1 -- 172.21.15.129:0/4001353659 >> v1:172.21.15.129:6789/0 conn(0x7f34381544b0 legacy=0x7f34380caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:37.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.385+0000 7f343d9ab640 1 -- 172.21.15.129:0/4001353659 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343815a9e0 con 0x7f343806c0b0 2024-03-23T15:54:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.387+0000 7f343d9ab640 1 --2- 172.21.15.129:0/4001353659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343806c0b0 0x7f34380c6000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f34380b3b50 tx=0x7f343002c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f65010402b963b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.392+0000 7f342ffff640 1 -- 172.21.15.129:0/4001353659 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343002cbb0 con 0x7f343806c0b0 2024-03-23T15:54:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.392+0000 7f342ffff640 1 -- 172.21.15.129:0/4001353659 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f343002cd10 con 0x7f343806c0b0 2024-03-23T15:54:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.392+0000 7f342ffff640 1 -- 172.21.15.129:0/4001353659 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430034610 con 0x7f343806c0b0 2024-03-23T15:54:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 -- 172.21.15.129:0/4001353659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343806c0b0 msgr2=0x7f34380c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 --2- 172.21.15.129:0/4001353659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343806c0b0 0x7f34380c6000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f34380b3b50 tx=0x7f343002c010 comp rx=0 tx=0).stop 2024-03-23T15:54:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 -- 172.21.15.129:0/4001353659 shutdown_connections 2024-03-23T15:54:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 --2- 172.21.15.129:0/4001353659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343806c0b0 0x7f34380c6000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 --2- 172.21.15.129:0/4001353659 >> v2:172.21.15.129:3300/0 conn(0x7f343815a2e0 0x7f343806bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 -- 172.21.15.129:0/4001353659 >> 172.21.15.129:0/4001353659 conn(0x7f3438061b50 msgr2=0x7f3438061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 -- 172.21.15.129:0/4001353659 shutdown_connections 2024-03-23T15:54:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.393+0000 7f343f40e640 1 -- 172.21.15.129:0/4001353659 wait complete. 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.394+0000 7f343f40e640 1 Processor -- start 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.394+0000 7f343f40e640 1 -- start start 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.395+0000 7f343f40e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 0x7f3438145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.395+0000 7f343f40e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34381544b0 0x7f34381f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.395+0000 7f343f40e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343815a2e0 0x7f343813eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.395+0000 7f343f40e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34381558b0 con 0x7f343806c0b0 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.395+0000 7f343f40e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34381f00d0 con 0x7f343815a2e0 2024-03-23T15:54:37.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.395+0000 7f343f40e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3438155010 con 0x7f34381544b0 2024-03-23T15:54:37.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.396+0000 7f343d1aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 0x7f3438145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:37.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.396+0000 7f343c9a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34381544b0 0x7f34381f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:37.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.396+0000 7f343d1aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 0x7f3438145ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40578/0 (socket says 172.21.15.129:40578) 2024-03-23T15:54:37.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.396+0000 7f343d1aa640 1 -- 172.21.15.129:0/2329914413 learned_addr learned my addr 172.21.15.129:0/2329914413 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:37.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.396+0000 7f343d9ab640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343815a2e0 0x7f343813eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:37.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.399+0000 7f343d1aa640 1 -- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34381544b0 msgr2=0x7f34381f0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:37.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.399+0000 7f343d1aa640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34381544b0 0x7f34381f0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.399+0000 7f343d1aa640 1 -- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343815a2e0 msgr2=0x7f343813eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:37.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.399+0000 7f343d1aa640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343815a2e0 0x7f343813eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.399+0000 7f343d1aa640 1 -- 172.21.15.129:0/2329914413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3430009b30 con 0x7f343806c0b0 2024-03-23T15:54:37.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.402+0000 7f343d1aa640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 0x7f3438145ae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f343400c400 tx=0x7f343400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:37.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.404+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340052d0 con 0x7f343806c0b0 2024-03-23T15:54:37.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.404+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34381552f0 con 0x7f343806c0b0 2024-03-23T15:54:37.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.404+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34381f0390 con 0x7f343806c0b0 2024-03-23T15:54:37.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.405+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438075740 con 0x7f343806c0b0 2024-03-23T15:54:37.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.614+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3434005430 con 0x7f343806c0b0 2024-03-23T15:54:37.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:54:37.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:54:37.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.689+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434007240 con 0x7f343806c0b0 2024-03-23T15:54:37.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.690+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3434007460 con 0x7f343806c0b0 2024-03-23T15:54:37.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.691+0000 7f342dffb640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f341c06b810 0x7f341c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:37.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.691+0000 7f343c9a9640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f341c06b810 0x7f341c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:37.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.692+0000 7f343c9a9640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f341c06b810 0x7f341c06dcd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34381595f0 tx=0x7f3428009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:37.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.715+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3434058270 con 0x7f343806c0b0 2024-03-23T15:54:37.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.715+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3434054510 con 0x7f343806c0b0 2024-03-23T15:54:37.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.939+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34380ca050 con 0x7f343806c0b0 2024-03-23T15:54:37.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.953+0000 7f342dffb640 1 -- 172.21.15.129:0/2329914413 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3434020020 con 0x7f343806c0b0 2024-03-23T15:54:37.956 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:37.956 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:37.957 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f341c06b810 msgr2=0x7f341c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f341c06b810 0x7f341c06dcd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34381595f0 tx=0x7f3428009190 comp rx=0 tx=0).stop 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 msgr2=0x7f3438145ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 0x7f3438145ae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f343400c400 tx=0x7f343400c8c0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 shutdown_connections 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f341c06b810 0x7f341c06dcd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f343815a2e0 0x7f343813eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34381544b0 0x7f34381f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 --2- 172.21.15.129:0/2329914413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f343806c0b0 0x7f3438145ae0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:37.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 >> 172.21.15.129:0/2329914413 conn(0x7f3438061b50 msgr2=0x7f34380c5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:37.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 shutdown_connections 2024-03-23T15:54:37.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:37.956+0000 7f343f40e640 1 -- 172.21.15.129:0/2329914413 wait complete. 2024-03-23T15:54:38.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:38.575 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:39.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:39.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.675+0000 7ffa657a7640 1 Processor -- start 2024-03-23T15:54:39.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.675+0000 7ffa657a7640 1 -- start start 2024-03-23T15:54:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.676+0000 7ffa657a7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffa601544b0 0x7ffa6015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.676+0000 7ffa657a7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 0x7ffa600c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.676+0000 7ffa657a7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa600cbcb0 con 0x7ffa600caa00 2024-03-23T15:54:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.676+0000 7ffa657a7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ffa600c5be0 con 0x7ffa601544b0 2024-03-23T15:54:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.676+0000 7ffa657a7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa600c69b0 con 0x7ffa600c6000 2024-03-23T15:54:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.677+0000 7ffa5f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 0x7ffa600c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.677+0000 7ffa5effd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ffa601544b0 0x7ffa6015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.677+0000 7ffa5f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 0x7ffa600c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:55116/0 (socket says 172.21.15.129:55116) 2024-03-23T15:54:39.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.677+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/2193481450 learned_addr learned my addr 172.21.15.129:0/2193481450 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:39.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.680+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/2193481450 >> v2:172.21.15.129:3300/0 conn(0x7ffa601544b0 msgr2=0x7ffa6015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:39.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.680+0000 7ffa5f7fe640 1 --2- 172.21.15.129:0/2193481450 >> v2:172.21.15.129:3300/0 conn(0x7ffa601544b0 0x7ffa6015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:39.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.680+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/2193481450 >> v1:172.21.15.129:6789/0 conn(0x7ffa600caa00 legacy=0x7ffa6006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:39.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.680+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/2193481450 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6015b280 con 0x7ffa600c6000 2024-03-23T15:54:39.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.680+0000 7ffa5effd640 1 --2- 172.21.15.129:0/2193481450 >> v2:172.21.15.129:3300/0 conn(0x7ffa601544b0 0x7ffa6015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:54:39.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.683+0000 7ffa5f7fe640 1 --2- 172.21.15.129:0/2193481450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 0x7ffa600c6470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009e50 tx=0x7ffa5802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4373cb3140741af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:39.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.686+0000 7ffa5dffb640 1 -- 172.21.15.129:0/2193481450 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5802cb90 con 0x7ffa600c6000 2024-03-23T15:54:39.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.688+0000 7ffa5dffb640 1 -- 172.21.15.129:0/2193481450 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5802ccf0 con 0x7ffa600c6000 2024-03-23T15:54:39.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.688+0000 7ffa5dffb640 1 -- 172.21.15.129:0/2193481450 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58034610 con 0x7ffa600c6000 2024-03-23T15:54:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 -- 172.21.15.129:0/2193481450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 msgr2=0x7ffa600c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 --2- 172.21.15.129:0/2193481450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 0x7ffa600c6470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009e50 tx=0x7ffa5802bff0 comp rx=0 tx=0).stop 2024-03-23T15:54:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 -- 172.21.15.129:0/2193481450 shutdown_connections 2024-03-23T15:54:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 --2- 172.21.15.129:0/2193481450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa600c6000 0x7ffa600c6470 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 --2- 172.21.15.129:0/2193481450 >> v2:172.21.15.129:3300/0 conn(0x7ffa601544b0 0x7ffa6015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:39.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 -- 172.21.15.129:0/2193481450 >> 172.21.15.129:0/2193481450 conn(0x7ffa60061b50 msgr2=0x7ffa60061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 -- 172.21.15.129:0/2193481450 shutdown_connections 2024-03-23T15:54:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.689+0000 7ffa657a7640 1 -- 172.21.15.129:0/2193481450 wait complete. 2024-03-23T15:54:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 Processor -- start 2024-03-23T15:54:39.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 -- start start 2024-03-23T15:54:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa600c6000 0x7ffa601f1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa600caa00 0x7ffa60157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 0x7ffa60141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa60156ef0 con 0x7ffa600caa00 2024-03-23T15:54:39.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa601592f0 con 0x7ffa600c6000 2024-03-23T15:54:39.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.691+0000 7ffa657a7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa60158130 con 0x7ffa601544b0 2024-03-23T15:54:39.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.692+0000 7ffa5f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 0x7ffa60141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:39.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.692+0000 7ffa5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa600caa00 0x7ffa60157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:39.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.692+0000 7ffa5effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa600c6000 0x7ffa601f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:39.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.692+0000 7ffa5f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 0x7ffa60141450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:55120/0 (socket says 172.21.15.129:55120) 2024-03-23T15:54:39.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.692+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/477700735 learned_addr learned my addr 172.21.15.129:0/477700735 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.696+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa600c6000 msgr2=0x7ffa601f1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.696+0000 7ffa5f7fe640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa600c6000 0x7ffa601f1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.696+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/477700735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa600caa00 msgr2=0x7ffa60157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.696+0000 7ffa5f7fe640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa600caa00 0x7ffa60157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.696+0000 7ffa5f7fe640 1 -- 172.21.15.129:0/477700735 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa50009400 con 0x7ffa601544b0 2024-03-23T15:54:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.696+0000 7ffa5e7fc640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa600caa00 0x7ffa60157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:54:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.698+0000 7ffa5f7fe640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 0x7ffa60141450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009e50 tx=0x7ffa58004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.705+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58009510 con 0x7ffa601544b0 2024-03-23T15:54:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.705+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5803e450 con 0x7ffa601544b0 2024-03-23T15:54:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.705+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa58009b30 con 0x7ffa601544b0 2024-03-23T15:54:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.705+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5803e800 con 0x7ffa601544b0 2024-03-23T15:54:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.705+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa601589c0 con 0x7ffa601544b0 2024-03-23T15:54:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.706+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa60075740 con 0x7ffa601544b0 2024-03-23T15:54:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.972+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffa5803e960 con 0x7ffa601544b0 2024-03-23T15:54:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.973+0000 7ffa477fe640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa4c06b890 0x7ffa4c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.973+0000 7ffa5effd640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa4c06b890 0x7ffa4c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.974+0000 7ffa5effd640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa4c06b890 0x7ffa4c06dd50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009c30 tx=0x7ffa50009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:39.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.982+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ffa5807b810 con 0x7ffa601544b0 2024-03-23T15:54:39.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:39.992+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa58077ab0 con 0x7ffa601544b0 2024-03-23T15:54:40.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.209+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa60159430 con 0x7ffa601544b0 2024-03-23T15:54:40.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.233+0000 7ffa477fe640 1 -- 172.21.15.129:0/477700735 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ffa5802ce60 con 0x7ffa601544b0 2024-03-23T15:54:40.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:40.237 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:40.237 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa4c06b890 msgr2=0x7ffa4c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa4c06b890 0x7ffa4c06dd50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009c30 tx=0x7ffa50009190 comp rx=0 tx=0).stop 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 msgr2=0x7ffa60141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 0x7ffa60141450 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009e50 tx=0x7ffa58004a40 comp rx=0 tx=0).stop 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 shutdown_connections 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa601544b0 0x7ffa60141450 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa600caa00 0x7ffa60157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa4c06b890 0x7ffa4c06dd50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 --2- 172.21.15.129:0/477700735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa600c6000 0x7ffa601f1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 >> 172.21.15.129:0/477700735 conn(0x7ffa60061b50 msgr2=0x7ffa6006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 shutdown_connections 2024-03-23T15:54:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:40.236+0000 7ffa657a7640 1 -- 172.21.15.129:0/477700735 wait complete. 2024-03-23T15:54:40.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:40.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:41.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:41.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.955+0000 7f8ba4daf640 1 Processor -- start 2024-03-23T15:54:41.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.955+0000 7f8ba4daf640 1 -- start start 2024-03-23T15:54:41.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.957+0000 7f8ba4daf640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8ba00ccd60 0x7f8ba015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.957+0000 7f8ba4daf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ba00cef40 0x7f8ba015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.957+0000 7f8ba4daf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba015b290 con 0x7f8ba00ceb00 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.957+0000 7f8ba4daf640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba00ce800 con 0x7f8ba00ccd60 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.957+0000 7f8ba4daf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba0154a70 con 0x7f8ba00cef40 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.957+0000 7f8b9ed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ba00cef40 0x7f8ba015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.958+0000 7f8b9e59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8ba00ccd60 0x7f8ba015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.958+0000 7f8b9ed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ba00cef40 0x7f8ba015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35118/0 (socket says 172.21.15.129:35118) 2024-03-23T15:54:41.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.958+0000 7f8b9ed9d640 1 -- 172.21.15.129:0/2796774679 learned_addr learned my addr 172.21.15.129:0/2796774679 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:41.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.960+0000 7f8b9ed9d640 1 -- 172.21.15.129:0/2796774679 >> v2:172.21.15.129:3300/0 conn(0x7f8ba00ccd60 msgr2=0x7f8ba015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:41.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.960+0000 7f8b9ed9d640 1 --2- 172.21.15.129:0/2796774679 >> v2:172.21.15.129:3300/0 conn(0x7f8ba00ccd60 0x7f8ba015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:41.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.960+0000 7f8b9ed9d640 1 -- 172.21.15.129:0/2796774679 >> v1:172.21.15.129:6789/0 conn(0x7f8ba00ceb00 legacy=0x7f8ba006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:41.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.960+0000 7f8b9ed9d640 1 -- 172.21.15.129:0/2796774679 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba00cd950 con 0x7f8ba00cef40 2024-03-23T15:54:41.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.962+0000 7f8b9ed9d640 1 --2- 172.21.15.129:0/2796774679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00cef40 0x7f8ba015b8f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00cf660 tx=0x7f8b9802c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31abdf254f07272 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:41.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.966+0000 7f8b9d59a640 1 -- 172.21.15.129:0/2796774679 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9802cbb0 con 0x7f8ba00cef40 2024-03-23T15:54:41.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.966+0000 7f8b9d59a640 1 -- 172.21.15.129:0/2796774679 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b9802cd10 con 0x7f8ba00cef40 2024-03-23T15:54:41.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.966+0000 7f8b9d59a640 1 -- 172.21.15.129:0/2796774679 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98034610 con 0x7f8ba00cef40 2024-03-23T15:54:41.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 -- 172.21.15.129:0/2796774679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00cef40 msgr2=0x7f8ba015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:41.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/2796774679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00cef40 0x7f8ba015b8f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00cf660 tx=0x7f8b9802c010 comp rx=0 tx=0).stop 2024-03-23T15:54:41.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 -- 172.21.15.129:0/2796774679 shutdown_connections 2024-03-23T15:54:41.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/2796774679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00cef40 0x7f8ba015b8f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:41.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/2796774679 >> v2:172.21.15.129:3300/0 conn(0x7f8ba00ccd60 0x7f8ba015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:41.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 -- 172.21.15.129:0/2796774679 >> 172.21.15.129:0/2796774679 conn(0x7f8ba00b1840 msgr2=0x7f8ba00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:41.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 -- 172.21.15.129:0/2796774679 shutdown_connections 2024-03-23T15:54:41.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.967+0000 7f8ba4daf640 1 -- 172.21.15.129:0/2796774679 wait complete. 2024-03-23T15:54:41.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.969+0000 7f8ba4daf640 1 Processor -- start 2024-03-23T15:54:41.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.970+0000 7f8ba4daf640 1 -- start start 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8ba4daf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00ccd60 0x7f8ba00c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8ba4daf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ba00ceb00 0x7f8ba01e4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8ba4daf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ba00cef40 0x7f8ba01e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8ba4daf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba01f0fa0 con 0x7f8ba00ceb00 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8ba4daf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba00c7a00 con 0x7f8ba00ccd60 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8ba4daf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba01f0130 con 0x7f8ba00cef40 2024-03-23T15:54:41.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8b9e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00ccd60 0x7f8ba00c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:41.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8b9dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ba00ceb00 0x7f8ba01e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:41.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8b9e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00ccd60 0x7f8ba00c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35126/0 (socket says 172.21.15.129:35126) 2024-03-23T15:54:41.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.971+0000 7f8b9e59c640 1 -- 172.21.15.129:0/4030080366 learned_addr learned my addr 172.21.15.129:0/4030080366 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:41.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.972+0000 7f8b9ed9d640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ba00cef40 0x7f8ba01e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:41.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.975+0000 7f8b9dd9b640 1 -- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ba00cef40 msgr2=0x7f8ba01e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:41.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.975+0000 7f8b9dd9b640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ba00cef40 0x7f8ba01e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:41.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.975+0000 7f8b9dd9b640 1 -- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00ccd60 msgr2=0x7f8ba00c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:41.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.975+0000 7f8b9dd9b640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00ccd60 0x7f8ba00c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:41.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.975+0000 7f8b9dd9b640 1 -- 172.21.15.129:0/4030080366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b98009b30 con 0x7f8ba00ceb00 2024-03-23T15:54:41.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.978+0000 7f8b9dd9b640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ba00ceb00 0x7f8ba01e4660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400c3c0 tx=0x7f8b9400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.982+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b940052d0 con 0x7f8ba00ceb00 2024-03-23T15:54:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.982+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b94005450 con 0x7f8ba00ceb00 2024-03-23T15:54:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.982+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba01f0320 con 0x7f8ba00ceb00 2024-03-23T15:54:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.982+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94007260 con 0x7f8ba00ceb00 2024-03-23T15:54:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.982+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba01f08e0 con 0x7f8ba00ceb00 2024-03-23T15:54:41.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:41.983+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba004f530 con 0x7f8ba00ceb00 2024-03-23T15:54:42.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.002+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b940073c0 con 0x7f8ba00ceb00 2024-03-23T15:54:42.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.003+0000 7f8b8effd640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b8006b810 0x7f8b8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:42.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.003+0000 7f8b9e59c640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b8006b810 0x7f8b8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:42.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.004+0000 7f8b9e59c640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b8006b810 0x7f8b8006dcd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009b60 tx=0x7f8b90009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.006+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8b94058300 con 0x7f8ba00ceb00 2024-03-23T15:54:42.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.251+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b940545a0 con 0x7f8ba00ceb00 2024-03-23T15:54:42.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.470+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ba00686f0 con 0x7f8ba00ceb00 2024-03-23T15:54:42.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.483+0000 7f8b8effd640 1 -- 172.21.15.129:0/4030080366 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8b94020020 con 0x7f8ba00ceb00 2024-03-23T15:54:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:42.487 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:42.487 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b8006b810 msgr2=0x7f8b8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b8006b810 0x7f8b8006dcd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009b60 tx=0x7f8b90009190 comp rx=0 tx=0).stop 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ba00ceb00 msgr2=0x7f8ba01e4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ba00ceb00 0x7f8ba01e4660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400c3c0 tx=0x7f8b9400c880 comp rx=0 tx=0).stop 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 shutdown_connections 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ba00cef40 0x7f8ba01e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ba00ceb00 0x7f8ba01e4660 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:42.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b8006b810 0x7f8b8006dcd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 --2- 172.21.15.129:0/4030080366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ba00ccd60 0x7f8ba00c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.485+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 >> 172.21.15.129:0/4030080366 conn(0x7f8ba00b1840 msgr2=0x7f8ba00ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.486+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 shutdown_connections 2024-03-23T15:54:42.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:42.486+0000 7f8ba4daf640 1 -- 172.21.15.129:0/4030080366 wait complete. 2024-03-23T15:54:43.104 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:43.104 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:44.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:44.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.204+0000 7f46ae69d640 1 Processor -- start 2024-03-23T15:54:44.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.204+0000 7f46ae69d640 1 -- start start 2024-03-23T15:54:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.205+0000 7f46ae69d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f46a815dba0 0x7f46a806cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.205+0000 7f46ae69d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a81540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.205+0000 7f46ae69d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a8070640 con 0x7f46a80cca30 2024-03-23T15:54:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.205+0000 7f46ae69d640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f46a80ccea0 con 0x7f46a815dba0 2024-03-23T15:54:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.205+0000 7f46ae69d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f46a80c4c00 con 0x7f46a815df80 2024-03-23T15:54:44.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.205+0000 7f46a7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f46a815dba0 0x7f46a806cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:44.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.206+0000 7f46acc3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a81540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:44.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.206+0000 7f46acc3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a81540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50632/0 (socket says 172.21.15.129:50632) 2024-03-23T15:54:44.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.206+0000 7f46acc3a640 1 -- 172.21.15.129:0/384853179 learned_addr learned my addr 172.21.15.129:0/384853179 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:44.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.208+0000 7f46acc3a640 1 -- 172.21.15.129:0/384853179 >> v2:172.21.15.129:3300/0 conn(0x7f46a815dba0 msgr2=0x7f46a806cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:44.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.208+0000 7f46acc3a640 1 --2- 172.21.15.129:0/384853179 >> v2:172.21.15.129:3300/0 conn(0x7f46a815dba0 0x7f46a806cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.208+0000 7f46acc3a640 1 -- 172.21.15.129:0/384853179 >> v1:172.21.15.129:6789/0 conn(0x7f46a80cca30 legacy=0x7f46a806ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:44.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.208+0000 7f46acc3a640 1 -- 172.21.15.129:0/384853179 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a806c240 con 0x7f46a815df80 2024-03-23T15:54:44.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.210+0000 7f46acc3a640 1 --2- 172.21.15.129:0/384853179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a81540a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009e80 tx=0x7f46a002bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42a788c2ff6a9c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:44.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.215+0000 7f46a6ffd640 1 -- 172.21.15.129:0/384853179 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a002cb70 con 0x7f46a815df80 2024-03-23T15:54:44.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.215+0000 7f46a6ffd640 1 -- 172.21.15.129:0/384853179 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a002ccd0 con 0x7f46a815df80 2024-03-23T15:54:44.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.215+0000 7f46a6ffd640 1 -- 172.21.15.129:0/384853179 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0034610 con 0x7f46a815df80 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 -- 172.21.15.129:0/384853179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 msgr2=0x7f46a81540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 --2- 172.21.15.129:0/384853179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a81540a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009e80 tx=0x7f46a002bfd0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 -- 172.21.15.129:0/384853179 shutdown_connections 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 --2- 172.21.15.129:0/384853179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a81540a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 --2- 172.21.15.129:0/384853179 >> v2:172.21.15.129:3300/0 conn(0x7f46a815dba0 0x7f46a806cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 -- 172.21.15.129:0/384853179 >> 172.21.15.129:0/384853179 conn(0x7f46a8071300 msgr2=0x7f46a80716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 -- 172.21.15.129:0/384853179 shutdown_connections 2024-03-23T15:54:44.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.216+0000 7f46ae69d640 1 -- 172.21.15.129:0/384853179 wait complete. 2024-03-23T15:54:44.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.217+0000 7f46ae69d640 1 Processor -- start 2024-03-23T15:54:44.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.217+0000 7f46ae69d640 1 -- start start 2024-03-23T15:54:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.218+0000 7f46ae69d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46a80cca30 0x7f46a81f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.218+0000 7f46ae69d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a815dba0 0x7f46a80c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.218+0000 7f46ae69d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a80ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.218+0000 7f46ae69d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8157880 con 0x7f46a815dba0 2024-03-23T15:54:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.218+0000 7f46ae69d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80c4da0 con 0x7f46a80cca30 2024-03-23T15:54:44.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.218+0000 7f46ae69d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f46a81f0c60 con 0x7f46a815df80 2024-03-23T15:54:44.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.219+0000 7f46acc3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a80ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:44.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.219+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a815dba0 0x7f46a80c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:44.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.219+0000 7f46a7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46a80cca30 0x7f46a81f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:44.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.219+0000 7f46acc3a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a80ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50644/0 (socket says 172.21.15.129:50644) 2024-03-23T15:54:44.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.219+0000 7f46acc3a640 1 -- 172.21.15.129:0/1693328734 learned_addr learned my addr 172.21.15.129:0/1693328734 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.222+0000 7f46acc3a640 1 -- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46a80cca30 msgr2=0x7f46a81f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.222+0000 7f46acc3a640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46a80cca30 0x7f46a81f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.222+0000 7f46acc3a640 1 -- 172.21.15.129:0/1693328734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a815dba0 msgr2=0x7f46a80c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.222+0000 7f46acc3a640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a815dba0 0x7f46a80c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.222+0000 7f46acc3a640 1 -- 172.21.15.129:0/1693328734 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a0009b30 con 0x7f46a815df80 2024-03-23T15:54:44.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.226+0000 7f46acc3a640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a80ca710 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009e80 tx=0x7f46a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:44.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.231+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0009510 con 0x7f46a815df80 2024-03-23T15:54:44.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.231+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a003e450 con 0x7f46a815df80 2024-03-23T15:54:44.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.231+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a80c7da0 con 0x7f46a815df80 2024-03-23T15:54:44.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.231+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a003e800 con 0x7f46a815df80 2024-03-23T15:54:44.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.231+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a80c52d0 con 0x7f46a815df80 2024-03-23T15:54:44.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.232+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a80686f0 con 0x7f46a815df80 2024-03-23T15:54:44.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.257+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f46a003e960 con 0x7f46a815df80 2024-03-23T15:54:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.257+0000 7f46a4ff9640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f469406b860 0x7f469406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.257+0000 7f46a7fff640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f469406b860 0x7f469406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.258+0000 7f46a7fff640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f469406b860 0x7f469406dd20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4698009b40 tx=0x7f4698009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:44.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.261+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f46a002ce40 con 0x7f46a815df80 2024-03-23T15:54:44.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.523+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f46a0077fe0 con 0x7f46a815df80 2024-03-23T15:54:44.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.782+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46a80c7650 con 0x7f46a815df80 2024-03-23T15:54:44.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.833+0000 7f46a4ff9640 1 -- 172.21.15.129:0/1693328734 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f46a007b6a0 con 0x7f46a815df80 2024-03-23T15:54:44.836 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:44.836 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:44.836 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:44.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f469406b860 msgr2=0x7f469406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f469406b860 0x7f469406dd20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4698009b40 tx=0x7f4698009190 comp rx=0 tx=0).stop 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 msgr2=0x7f46a80ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a80ca710 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009e80 tx=0x7f46a0004a40 comp rx=0 tx=0).stop 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 shutdown_connections 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46a815df80 0x7f46a80ca710 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46a815dba0 0x7f46a80c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f469406b860 0x7f469406dd20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 --2- 172.21.15.129:0/1693328734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46a80cca30 0x7f46a81f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.835+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 >> 172.21.15.129:0/1693328734 conn(0x7f46a8071300 msgr2=0x7f46a80cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.836+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 shutdown_connections 2024-03-23T15:54:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:44.836+0000 7f46ae69d640 1 -- 172.21.15.129:0/1693328734 wait complete. 2024-03-23T15:54:45.454 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:45.454 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:46.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:46.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.554+0000 7f9a3fa8e640 1 Processor -- start 2024-03-23T15:54:46.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.554+0000 7f9a3fa8e640 1 -- start start 2024-03-23T15:54:46.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3fa8e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a3806c030 0x7f9a3806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:46.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3fa8e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9a380cc080 0x7f9a3815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:46.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3fa8e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a3806cae0 con 0x7f9a380cbc60 2024-03-23T15:54:46.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3fa8e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9a380cabc0 con 0x7f9a380cc080 2024-03-23T15:54:46.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3fa8e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a380caea0 con 0x7f9a3806c030 2024-03-23T15:54:46.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3e02b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9a380cc080 0x7f9a3815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:46.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.555+0000 7f9a3d82a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a3806c030 0x7f9a3806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:46.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.556+0000 7f9a3e02b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9a380cc080 0x7f9a3815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36144/0 (socket says 172.21.15.129:36144) 2024-03-23T15:54:46.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.556+0000 7f9a3e02b640 1 -- 172.21.15.129:0/2128721830 learned_addr learned my addr 172.21.15.129:0/2128721830 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.559+0000 7f9a3e02b640 1 -- 172.21.15.129:0/2128721830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a3806c030 msgr2=0x7f9a3806c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.559+0000 7f9a3e02b640 1 --2- 172.21.15.129:0/2128721830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a3806c030 0x7f9a3806c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.559+0000 7f9a3e02b640 1 -- 172.21.15.129:0/2128721830 >> v1:172.21.15.129:6789/0 conn(0x7f9a380cbc60 legacy=0x7f9a3815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:46.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.559+0000 7f9a3e02b640 1 -- 172.21.15.129:0/2128721830 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a380cb180 con 0x7f9a380cc080 2024-03-23T15:54:46.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.565+0000 7f9a3e02b640 1 --2- 172.21.15.129:0/2128721830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a3815e010 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009e50 tx=0x7f9a3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151cbb2c4e1637ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.570+0000 7f9a3c828640 1 -- 172.21.15.129:0/2128721830 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3002ca70 con 0x7f9a380cc080 2024-03-23T15:54:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.570+0000 7f9a3c828640 1 -- 172.21.15.129:0/2128721830 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3002cbd0 con 0x7f9a380cc080 2024-03-23T15:54:46.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.570+0000 7f9a3c828640 1 -- 172.21.15.129:0/2128721830 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a300343f0 con 0x7f9a380cc080 2024-03-23T15:54:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/2128721830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 msgr2=0x7f9a3815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/2128721830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a3815e010 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009e50 tx=0x7f9a3002bed0 comp rx=0 tx=0).stop 2024-03-23T15:54:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/2128721830 shutdown_connections 2024-03-23T15:54:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/2128721830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a3815e010 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/2128721830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a3806c030 0x7f9a3806c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:46.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/2128721830 >> 172.21.15.129:0/2128721830 conn(0x7f9a380cfb20 msgr2=0x7f9a380cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:46.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/2128721830 shutdown_connections 2024-03-23T15:54:46.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.571+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/2128721830 wait complete. 2024-03-23T15:54:46.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 Processor -- start 2024-03-23T15:54:46.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 -- start start 2024-03-23T15:54:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a3806c030 0x7f9a381f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a380cbc60 0x7f9a380b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a380b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a380b4f90 con 0x7f9a380cc080 2024-03-23T15:54:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381578c0 con 0x7f9a3806c030 2024-03-23T15:54:46.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.573+0000 7f9a3fa8e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a381f42b0 con 0x7f9a380cbc60 2024-03-23T15:54:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.574+0000 7f9a3e02b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a380b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.574+0000 7f9a3d029640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a380cbc60 0x7f9a380b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:46.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.574+0000 7f9a3d82a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a3806c030 0x7f9a381f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:46.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.574+0000 7f9a3e02b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a380b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36160/0 (socket says 172.21.15.129:36160) 2024-03-23T15:54:46.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.574+0000 7f9a3e02b640 1 -- 172.21.15.129:0/3574910717 learned_addr learned my addr 172.21.15.129:0/3574910717 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:46.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.579+0000 7f9a3e02b640 1 -- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a380cbc60 msgr2=0x7f9a380b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:46.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.579+0000 7f9a3e02b640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a380cbc60 0x7f9a380b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:46.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.579+0000 7f9a3e02b640 1 -- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a3806c030 msgr2=0x7f9a381f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:46.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.579+0000 7f9a3e02b640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a3806c030 0x7f9a381f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:46.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.579+0000 7f9a3e02b640 1 -- 172.21.15.129:0/3574910717 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a30009b30 con 0x7f9a380cc080 2024-03-23T15:54:46.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.582+0000 7f9a3e02b640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a380b3d00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009e50 tx=0x7f9a30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.587+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30009510 con 0x7f9a380cc080 2024-03-23T15:54:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.587+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3003e450 con 0x7f9a380cc080 2024-03-23T15:54:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.587+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3003e800 con 0x7f9a380cc080 2024-03-23T15:54:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.587+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a38157600 con 0x7f9a380cc080 2024-03-23T15:54:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.587+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a381f3fd0 con 0x7f9a380cc080 2024-03-23T15:54:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.588+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a38075740 con 0x7f9a380cc080 2024-03-23T15:54:46.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.607+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9a3003e960 con 0x7f9a380cc080 2024-03-23T15:54:46.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.607+0000 7f9a2e7fc640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9a1c06b860 0x7f9a1c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:46.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.607+0000 7f9a3d82a640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9a1c06b860 0x7f9a1c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:46.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.608+0000 7f9a3d82a640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9a1c06b860 0x7f9a1c06dd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009aa0 tx=0x7f9a34009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:46.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.865+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9a3007bbf0 con 0x7f9a380cc080 2024-03-23T15:54:46.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:46.865+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9a30077e90 con 0x7f9a380cc080 2024-03-23T15:54:47.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.098+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a38153e70 con 0x7f9a380cc080 2024-03-23T15:54:47.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.119+0000 7f9a2e7fc640 1 -- 172.21.15.129:0/3574910717 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9a3007b550 con 0x7f9a380cc080 2024-03-23T15:54:47.122 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:47.123 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:47.123 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9a1c06b860 msgr2=0x7f9a1c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9a1c06b860 0x7f9a1c06dd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009aa0 tx=0x7f9a34009190 comp rx=0 tx=0).stop 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 msgr2=0x7f9a380b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a380b3d00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009e50 tx=0x7f9a30004a40 comp rx=0 tx=0).stop 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 shutdown_connections 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a380cc080 0x7f9a380b3d00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a380cbc60 0x7f9a380b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9a1c06b860 0x7f9a1c06dd20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 --2- 172.21.15.129:0/3574910717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a3806c030 0x7f9a381f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 >> 172.21.15.129:0/3574910717 conn(0x7f9a380cfb20 msgr2=0x7f9a380cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:47.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 shutdown_connections 2024-03-23T15:54:47.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:47.122+0000 7f9a3fa8e640 1 -- 172.21.15.129:0/3574910717 wait complete. 2024-03-23T15:54:47.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:47.742 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:48.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:48.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.844+0000 7fd0e220b640 1 Processor -- start 2024-03-23T15:54:48.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.844+0000 7fd0e220b640 1 -- start start 2024-03-23T15:54:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.844+0000 7fd0e220b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd0dc15dba0 0x7fd0dc06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.844+0000 7fd0e220b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc15df80 0x7fd0dc1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.844+0000 7fd0e220b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0dc070640 con 0x7fd0dc0cca30 2024-03-23T15:54:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.845+0000 7fd0e220b640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd0dc0ccea0 con 0x7fd0dc15dba0 2024-03-23T15:54:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.845+0000 7fd0e220b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc0c4c00 con 0x7fd0dc15df80 2024-03-23T15:54:48.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.845+0000 7fd0db7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd0dc15dba0 0x7fd0dc06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:48.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.845+0000 7fd0dbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc15df80 0x7fd0dc1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:48.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.846+0000 7fd0db7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd0dc15dba0 0x7fd0dc06cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36176/0 (socket says 172.21.15.129:36176) 2024-03-23T15:54:48.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.846+0000 7fd0db7fe640 1 -- 172.21.15.129:0/3204083205 learned_addr learned my addr 172.21.15.129:0/3204083205 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:48.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.849+0000 7fd0db7fe640 1 -- 172.21.15.129:0/3204083205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc15df80 msgr2=0x7fd0dc1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.849+0000 7fd0db7fe640 1 --2- 172.21.15.129:0/3204083205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc15df80 0x7fd0dc1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.849+0000 7fd0db7fe640 1 -- 172.21.15.129:0/3204083205 >> v1:172.21.15.129:6789/0 conn(0x7fd0dc0cca30 legacy=0x7fd0dc06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.849+0000 7fd0db7fe640 1 -- 172.21.15.129:0/3204083205 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc06c240 con 0x7fd0dc15dba0 2024-03-23T15:54:48.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.852+0000 7fd0db7fe640 1 --2- 172.21.15.129:0/3204083205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc06cf70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0098a0 tx=0x7fd0cc02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362900ab23326d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.855+0000 7fd0da7fc640 1 -- 172.21.15.129:0/3204083205 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc02f830 con 0x7fd0dc15dba0 2024-03-23T15:54:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.856+0000 7fd0da7fc640 1 -- 172.21.15.129:0/3204083205 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0cc02f990 con 0x7fd0dc15dba0 2024-03-23T15:54:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.857+0000 7fd0da7fc640 1 -- 172.21.15.129:0/3204083205 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc02fd20 con 0x7fd0dc15dba0 2024-03-23T15:54:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.857+0000 7fd0e220b640 1 -- 172.21.15.129:0/3204083205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15dba0 msgr2=0x7fd0dc06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.857+0000 7fd0e220b640 1 --2- 172.21.15.129:0/3204083205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc06cf70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0098a0 tx=0x7fd0cc02ba10 comp rx=0 tx=0).stop 2024-03-23T15:54:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.857+0000 7fd0e220b640 1 -- 172.21.15.129:0/3204083205 shutdown_connections 2024-03-23T15:54:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.857+0000 7fd0e220b640 1 --2- 172.21.15.129:0/3204083205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc15df80 0x7fd0dc1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:48.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.857+0000 7fd0e220b640 1 --2- 172.21.15.129:0/3204083205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc06cf70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:48.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.858+0000 7fd0e220b640 1 -- 172.21.15.129:0/3204083205 >> 172.21.15.129:0/3204083205 conn(0x7fd0dc071300 msgr2=0x7fd0dc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:48.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.858+0000 7fd0e220b640 1 -- 172.21.15.129:0/3204083205 shutdown_connections 2024-03-23T15:54:48.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.858+0000 7fd0e220b640 1 -- 172.21.15.129:0/3204083205 wait complete. 2024-03-23T15:54:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.859+0000 7fd0e220b640 1 Processor -- start 2024-03-23T15:54:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.860+0000 7fd0e220b640 1 -- start start 2024-03-23T15:54:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.861+0000 7fd0e220b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc0cca30 0x7fd0dc153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.861+0000 7fd0e220b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.861+0000 7fd0e220b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15df80 0x7fd0dc147db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.861+0000 7fd0e220b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1462c0 con 0x7fd0dc15df80 2024-03-23T15:54:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.861+0000 7fd0e220b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1e4ee0 con 0x7fd0dc15dba0 2024-03-23T15:54:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.861+0000 7fd0e220b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc1e51c0 con 0x7fd0dc0cca30 2024-03-23T15:54:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.862+0000 7fd0db7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc0cca30 0x7fd0dc153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.862+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.862+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35136/0 (socket says 172.21.15.129:35136) 2024-03-23T15:54:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.862+0000 7fd0daffd640 1 -- 172.21.15.129:0/2548856359 learned_addr learned my addr 172.21.15.129:0/2548856359 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:48.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.865+0000 7fd0daffd640 1 -- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc0cca30 msgr2=0x7fd0dc153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:48.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.865+0000 7fd0daffd640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc0cca30 0x7fd0dc153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:48.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.865+0000 7fd0daffd640 1 -- 172.21.15.129:0/2548856359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15df80 msgr2=0x7fd0dc147db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:48.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.865+0000 7fd0daffd640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15df80 0x7fd0dc147db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:48.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.865+0000 7fd0daffd640 1 -- 172.21.15.129:0/2548856359 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc0094d0 con 0x7fd0dc15dba0 2024-03-23T15:54:48.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.868+0000 7fd0daffd640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc147870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c430 tx=0x7fd0d000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:48.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.873+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d00035f0 con 0x7fd0dc15dba0 2024-03-23T15:54:48.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.873+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d0003750 con 0x7fd0dc15dba0 2024-03-23T15:54:48.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.873+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc147070 con 0x7fd0dc15dba0 2024-03-23T15:54:48.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.873+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0003b40 con 0x7fd0dc15dba0 2024-03-23T15:54:48.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.873+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc146510 con 0x7fd0dc15dba0 2024-03-23T15:54:48.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.874+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0dc0686f0 con 0x7fd0dc15dba0 2024-03-23T15:54:48.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.900+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd0d0003ca0 con 0x7fd0dc15dba0 2024-03-23T15:54:48.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.901+0000 7fd0c3fff640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd0c806b840 0x7fd0c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:48.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.901+0000 7fd0db7fe640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd0c806b840 0x7fd0c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:48.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:48.902+0000 7fd0db7fe640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd0c806b840 0x7fd0c806dd00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc036000 tx=0x7fd0cc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:49.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.165+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd0d00582e0 con 0x7fd0dc15dba0 2024-03-23T15:54:49.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.184+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0d0054580 con 0x7fd0dc15dba0 2024-03-23T15:54:49.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.411+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0dc147530 con 0x7fd0dc15dba0 2024-03-23T15:54:49.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.448+0000 7fd0c3fff640 1 -- 172.21.15.129:0/2548856359 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd0d0017070 con 0x7fd0dc15dba0 2024-03-23T15:54:49.451 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:49.451 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:49.452 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:49.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd0c806b840 msgr2=0x7fd0c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:49.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd0c806b840 0x7fd0c806dd00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc036000 tx=0x7fd0cc008030 comp rx=0 tx=0).stop 2024-03-23T15:54:49.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 msgr2=0x7fd0dc147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:49.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc147870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c430 tx=0x7fd0d000c8f0 comp rx=0 tx=0).stop 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 shutdown_connections 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0dc15df80 0x7fd0dc147db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd0c806b840 0x7fd0c806dd00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0dc15dba0 0x7fd0dc147870 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 --2- 172.21.15.129:0/2548856359 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0dc0cca30 0x7fd0dc153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 >> 172.21.15.129:0/2548856359 conn(0x7fd0dc071300 msgr2=0x7fd0dc0c97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 shutdown_connections 2024-03-23T15:54:49.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:49.450+0000 7fd0e220b640 1 -- 172.21.15.129:0/2548856359 wait complete. 2024-03-23T15:54:50.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:50.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:51.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:51.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.170+0000 7fadbabe2640 1 Processor -- start 2024-03-23T15:54:51.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.170+0000 7fadbabe2640 1 -- start start 2024-03-23T15:54:51.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadbabe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb406c030 0x7fadb406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:51.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadbabe2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fadb40cc080 0x7fadb415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:51.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadbabe2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadb406cae0 con 0x7fadb40cbc60 2024-03-23T15:54:51.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadbabe2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fadb40cabc0 con 0x7fadb40cc080 2024-03-23T15:54:51.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadbabe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fadb40caea0 con 0x7fadb406c030 2024-03-23T15:54:51.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadb917f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fadb40cc080 0x7fadb415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:51.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.171+0000 7fadb897e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb406c030 0x7fadb406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:51.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.172+0000 7fadb3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fadb40cbc60 0x7fadb415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:58574/0 (socket says 172.21.15.129:58574) 2024-03-23T15:54:51.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.172+0000 7fadb3fff640 1 -- 172.21.15.129:0/3379767999 learned_addr learned my addr 172.21.15.129:0/3379767999 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.175+0000 7fadb897e640 1 -- 172.21.15.129:0/3379767999 >> v2:172.21.15.157:3300/0 conn(0x7fadb40cc080 msgr2=0x7fadb415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.175+0000 7fadb897e640 1 --2- 172.21.15.129:0/3379767999 >> v2:172.21.15.157:3300/0 conn(0x7fadb40cc080 0x7fadb415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.175+0000 7fadb897e640 1 -- 172.21.15.129:0/3379767999 >> v1:172.21.15.157:6789/0 conn(0x7fadb40cbc60 legacy=0x7fadb415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.175+0000 7fadb897e640 1 -- 172.21.15.129:0/3379767999 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb40cb180 con 0x7fadb406c030 2024-03-23T15:54:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.178+0000 7fadb897e640 1 --2- 172.21.15.129:0/3379767999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb406c030 0x7fadb406c410 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fada80097a0 tx=0x7fada802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7775b7b1c7aa9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:51.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadb37fe640 1 -- 172.21.15.129:0/3379767999 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada802e070 con 0x7fadb406c030 2024-03-23T15:54:51.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadb37fe640 1 -- 172.21.15.129:0/3379767999 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada80315a0 con 0x7fadb406c030 2024-03-23T15:54:51.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadb37fe640 1 -- 172.21.15.129:0/3379767999 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8031930 con 0x7fadb406c030 2024-03-23T15:54:51.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 -- 172.21.15.129:0/3379767999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb406c030 msgr2=0x7fadb406c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:51.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3379767999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb406c030 0x7fadb406c410 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fada80097a0 tx=0x7fada802b840 comp rx=0 tx=0).stop 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 -- 172.21.15.129:0/3379767999 shutdown_connections 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3379767999 >> v2:172.21.15.157:3300/0 conn(0x7fadb40cc080 0x7fadb415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3379767999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb406c030 0x7fadb406c410 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 -- 172.21.15.129:0/3379767999 >> 172.21.15.129:0/3379767999 conn(0x7fadb40cfb20 msgr2=0x7fadb40cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 -- 172.21.15.129:0/3379767999 shutdown_connections 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.184+0000 7fadbabe2640 1 -- 172.21.15.129:0/3379767999 wait complete. 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.186+0000 7fadbabe2640 1 Processor -- start 2024-03-23T15:54:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.186+0000 7fadbabe2640 1 -- start start 2024-03-23T15:54:51.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.186+0000 7fadbabe2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadb406c030 0x7fadb41f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:51.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.186+0000 7fadbabe2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 0x7fadb415dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:51.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.186+0000 7fadbabe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb40cc080 0x7fadb414ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:51.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadbabe2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fadb40c4070 con 0x7fadb406c030 2024-03-23T15:54:51.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadbabe2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fadb41515d0 con 0x7fadb40cbc60 2024-03-23T15:54:51.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadbabe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fadb41f0e80 con 0x7fadb40cc080 2024-03-23T15:54:51.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadb3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 0x7fadb415dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:51.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadb917f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb40cc080 0x7fadb414ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:51.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadb3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 0x7fadb415dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35860/0 (socket says 172.21.15.129:35860) 2024-03-23T15:54:51.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.187+0000 7fadb3fff640 1 -- 172.21.15.129:0/3734844696 learned_addr learned my addr 172.21.15.129:0/3734844696 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:51.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.191+0000 7fadb897e640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadb406c030 0x7fadb41f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:51.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.191+0000 7fadb3fff640 1 -- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb40cc080 msgr2=0x7fadb414ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:51.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.191+0000 7fadb3fff640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb40cc080 0x7fadb414ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.191+0000 7fadb3fff640 1 -- 172.21.15.129:0/3734844696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadb406c030 msgr2=0x7fadb41f0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:51.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.191+0000 7fadb3fff640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadb406c030 0x7fadb41f0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.191+0000 7fadb3fff640 1 -- 172.21.15.129:0/3734844696 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada8009400 con 0x7fadb40cbc60 2024-03-23T15:54:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.194+0000 7fadb3fff640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 0x7fadb415dfb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fada000c4d0 tx=0x7fada000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.203+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0005ce0 con 0x7fadb40cbc60 2024-03-23T15:54:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.203+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada0005e40 con 0x7fadb40cbc60 2024-03-23T15:54:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.203+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0007270 con 0x7fadb40cbc60 2024-03-23T15:54:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.203+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb41506e0 con 0x7fadb40cbc60 2024-03-23T15:54:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.203+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb4150ca0 con 0x7fadb40cbc60 2024-03-23T15:54:51.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.204+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb414e9d0 con 0x7fadb40cbc60 2024-03-23T15:54:51.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.234+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fada00073d0 con 0x7fadb40cbc60 2024-03-23T15:54:51.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.235+0000 7fadb17fa640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fad9806b810 0x7fad9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:51.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.235+0000 7fadb897e640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fad9806b810 0x7fad9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:51.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.236+0000 7fadb897e640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fad9806b810 0x7fad9806dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fada8036000 tx=0x7fada803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:51.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.240+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fada00582f0 con 0x7fadb40cbc60 2024-03-23T15:54:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.496+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fada0054590 con 0x7fadb40cbc60 2024-03-23T15:54:51.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.731+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadb4075740 con 0x7fadb40cbc60 2024-03-23T15:54:51.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.752+0000 7fadb17fa640 1 -- 172.21.15.129:0/3734844696 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fada0017090 con 0x7fadb40cbc60 2024-03-23T15:54:51.755 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:51.755 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:51.755 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:51.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fad9806b810 msgr2=0x7fad9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:51.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fad9806b810 0x7fad9806dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fada8036000 tx=0x7fada803d040 comp rx=0 tx=0).stop 2024-03-23T15:54:51.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 msgr2=0x7fadb415dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:51.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 0x7fadb415dfb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fada000c4d0 tx=0x7fada000c990 comp rx=0 tx=0).stop 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 shutdown_connections 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fadb40cc080 0x7fadb414ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fad9806b810 0x7fad9806dcd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadb40cbc60 0x7fadb415dfb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 --2- 172.21.15.129:0/3734844696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadb406c030 0x7fadb41f0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 >> 172.21.15.129:0/3734844696 conn(0x7fadb40cfb20 msgr2=0x7fadb40ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.754+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 shutdown_connections 2024-03-23T15:54:51.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:51.755+0000 7fadbabe2640 1 -- 172.21.15.129:0/3734844696 wait complete. 2024-03-23T15:54:52.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:52.372 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:53.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:53.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.471+0000 7f775b53e640 1 Processor -- start 2024-03-23T15:54:53.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.471+0000 7f775b53e640 1 -- start start 2024-03-23T15:54:53.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.473+0000 7f775b53e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f775406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:53.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.473+0000 7f775b53e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77540cbcb0 con 0x7f77541544b0 2024-03-23T15:54:53.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.473+0000 7f775b53e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775415b280 con 0x7f775406c0b0 2024-03-23T15:54:53.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.473+0000 7f775b53e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77540c5be0 con 0x7f775415a2e0 2024-03-23T15:54:53.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.473+0000 7f7758ad9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f775406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:53.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.474+0000 7f7759adb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f775406c0b0 0x7f77540c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:58582/0 (socket says 172.21.15.129:58582) 2024-03-23T15:54:53.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.474+0000 7f77592da640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f77541544b0 0x7f77540caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60958/0 (socket says 172.21.15.129:60958) 2024-03-23T15:54:53.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.474+0000 7f7759adb640 1 -- 172.21.15.129:0/2828470820 learned_addr learned my addr 172.21.15.129:0/2828470820 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:53.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.477+0000 7f774bfff640 1 -- 172.21.15.129:0/2828470820 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065649876 0 0) 0x7f77540cbcb0 con 0x7f77541544b0 2024-03-23T15:54:53.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.477+0000 7f7758ad9640 1 -- 172.21.15.129:0/2828470820 >> v1:172.21.15.157:6789/0 conn(0x7f775406c0b0 legacy=0x7f77540c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:53.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.477+0000 7f7758ad9640 1 -- 172.21.15.129:0/2828470820 >> v1:172.21.15.129:6789/0 conn(0x7f77541544b0 legacy=0x7f77540caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:53.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.477+0000 7f7758ad9640 1 -- 172.21.15.129:0/2828470820 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775415a9e0 con 0x7f775415a2e0 2024-03-23T15:54:53.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.479+0000 7f7758ad9640 1 --2- 172.21.15.129:0/2828470820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f775406bb70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f77440097d0 tx=0x7f774402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45a4c5fc72f2be86 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f774bfff640 1 -- 172.21.15.129:0/2828470820 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744030880 con 0x7f775415a2e0 2024-03-23T15:54:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f774bfff640 1 -- 172.21.15.129:0/2828470820 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77440309e0 con 0x7f775415a2e0 2024-03-23T15:54:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f774bfff640 1 -- 172.21.15.129:0/2828470820 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744030d70 con 0x7f775415a2e0 2024-03-23T15:54:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 -- 172.21.15.129:0/2828470820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 msgr2=0x7f775406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 --2- 172.21.15.129:0/2828470820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f775406bb70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f77440097d0 tx=0x7f774402b9c0 comp rx=0 tx=0).stop 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 -- 172.21.15.129:0/2828470820 shutdown_connections 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 --2- 172.21.15.129:0/2828470820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f775406bb70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 -- 172.21.15.129:0/2828470820 >> 172.21.15.129:0/2828470820 conn(0x7f7754061b50 msgr2=0x7f7754061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 -- 172.21.15.129:0/2828470820 shutdown_connections 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.484+0000 7f775b53e640 1 -- 172.21.15.129:0/2828470820 wait complete. 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 Processor -- start 2024-03-23T15:54:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 -- start start 2024-03-23T15:54:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 0x7f7754146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77541544b0 0x7f7754141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f7754141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7754157b50 con 0x7f77541544b0 2024-03-23T15:54:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7754155d60 con 0x7f775406c0b0 2024-03-23T15:54:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.486+0000 7f775b53e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7754157600 con 0x7f775415a2e0 2024-03-23T15:54:53.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.487+0000 7f7758ad9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77541544b0 0x7f7754141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:53.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.487+0000 7f77592da640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 0x7f7754146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:53.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.487+0000 7f77592da640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 0x7f7754146d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35874/0 (socket says 172.21.15.129:35874) 2024-03-23T15:54:53.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.487+0000 7f77592da640 1 -- 172.21.15.129:0/2572870021 learned_addr learned my addr 172.21.15.129:0/2572870021 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:53.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.488+0000 7f7759adb640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f7754141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:53.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.492+0000 7f77592da640 1 -- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 msgr2=0x7f7754141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:53.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.492+0000 7f77592da640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f7754141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:53.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.492+0000 7f77592da640 1 -- 172.21.15.129:0/2572870021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77541544b0 msgr2=0x7f7754141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:53.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.492+0000 7f77592da640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77541544b0 0x7f7754141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:53.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.492+0000 7f77592da640 1 -- 172.21.15.129:0/2572870021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7744009400 con 0x7f775406c0b0 2024-03-23T15:54:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.496+0000 7f77592da640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 0x7f7754146d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f775000c500 tx=0x7f775000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:53.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.501+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750007560 con 0x7f775406c0b0 2024-03-23T15:54:53.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.501+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77500047e0 con 0x7f775406c0b0 2024-03-23T15:54:53.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.501+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750004b90 con 0x7f775406c0b0 2024-03-23T15:54:53.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.501+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7754141dd0 con 0x7f775406c0b0 2024-03-23T15:54:53.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.501+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7754155710 con 0x7f775406c0b0 2024-03-23T15:54:53.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.502+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77540b3450 con 0x7f775406c0b0 2024-03-23T15:54:53.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.532+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f775001c3b0 con 0x7f775406c0b0 2024-03-23T15:54:53.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.532+0000 7f7749ffb640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f773806b860 0x7f773806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:53.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.533+0000 7f7758ad9640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f773806b860 0x7f773806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:53.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.533+0000 7f7758ad9640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f773806b860 0x7f773806dd20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7744036000 tx=0x7f7744008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:53.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.538+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7750058490 con 0x7f775406c0b0 2024-03-23T15:54:53.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:53.799+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7750054730 con 0x7f775406c0b0 2024-03-23T15:54:54.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.020+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77540b3130 con 0x7f775406c0b0 2024-03-23T15:54:54.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.042+0000 7f7749ffb640 1 -- 172.21.15.129:0/2572870021 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f7750018020 con 0x7f775406c0b0 2024-03-23T15:54:54.045 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:54.045 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:54.046 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f773806b860 msgr2=0x7f773806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f773806b860 0x7f773806dd20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7744036000 tx=0x7f7744008030 comp rx=0 tx=0).stop 2024-03-23T15:54:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 msgr2=0x7f7754146d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 0x7f7754146d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f775000c500 tx=0x7f775000c9c0 comp rx=0 tx=0).stop 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 shutdown_connections 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f775415a2e0 0x7f7754141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77541544b0 0x7f7754141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f773806b860 0x7f773806dd20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 --2- 172.21.15.129:0/2572870021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f775406c0b0 0x7f7754146d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 >> 172.21.15.129:0/2572870021 conn(0x7f7754061b50 msgr2=0x7f7754151b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 shutdown_connections 2024-03-23T15:54:54.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:54.044+0000 7f775b53e640 1 -- 172.21.15.129:0/2572870021 wait complete. 2024-03-23T15:54:54.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:54.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:55.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.764+0000 7fe62a53a640 1 Processor -- start 2024-03-23T15:54:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.764+0000 7fe62a53a640 1 -- start start 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.765+0000 7fe62a53a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 0x7fe62406cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.765+0000 7fe62a53a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe62415e040 0x7fe6240cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.765+0000 7fe62a53a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62406d2c0 con 0x7fe62406f570 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.765+0000 7fe62a53a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe62415e740 con 0x7fe62415e040 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.765+0000 7fe62a53a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe6240ce2e0 con 0x7fe62415dc60 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.766+0000 7fe6237fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe62415e040 0x7fe6240cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.766+0000 7fe623fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 0x7fe62406cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.766+0000 7fe628ad7640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fe62406f570 0x7fe6240cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60968/0 (socket says 172.21.15.129:60968) 2024-03-23T15:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.766+0000 7fe628ad7640 1 -- 172.21.15.129:0/4008699907 learned_addr learned my addr 172.21.15.129:0/4008699907 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.766+0000 7fe623fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 0x7fe62406cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:45832/0 (socket says 172.21.15.129:45832) 2024-03-23T15:54:55.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.768+0000 7fe623fff640 1 -- 172.21.15.129:0/4008699907 >> v2:172.21.15.129:3300/0 conn(0x7fe62415e040 msgr2=0x7fe6240cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:55.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.768+0000 7fe623fff640 1 --2- 172.21.15.129:0/4008699907 >> v2:172.21.15.129:3300/0 conn(0x7fe62415e040 0x7fe6240cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:55.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.768+0000 7fe623fff640 1 -- 172.21.15.129:0/4008699907 >> v1:172.21.15.129:6789/0 conn(0x7fe62406f570 legacy=0x7fe6240cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:55.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.768+0000 7fe623fff640 1 -- 172.21.15.129:0/4008699907 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6240ccd00 con 0x7fe62415dc60 2024-03-23T15:54:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.771+0000 7fe623fff640 1 --2- 172.21.15.129:0/4008699907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 0x7fe62406cc50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe6140097a0 tx=0x7fe61402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b422d0f9e30422e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.775+0000 7fe622ffd640 1 -- 172.21.15.129:0/4008699907 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61402e070 con 0x7fe62415dc60 2024-03-23T15:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.775+0000 7fe622ffd640 1 -- 172.21.15.129:0/4008699907 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6140315a0 con 0x7fe62415dc60 2024-03-23T15:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.775+0000 7fe622ffd640 1 -- 172.21.15.129:0/4008699907 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614031930 con 0x7fe62415dc60 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 -- 172.21.15.129:0/4008699907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 msgr2=0x7fe62406cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 --2- 172.21.15.129:0/4008699907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 0x7fe62406cc50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe6140097a0 tx=0x7fe61402b840 comp rx=0 tx=0).stop 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 -- 172.21.15.129:0/4008699907 shutdown_connections 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 --2- 172.21.15.129:0/4008699907 >> v2:172.21.15.129:3300/0 conn(0x7fe62415e040 0x7fe6240cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 --2- 172.21.15.129:0/4008699907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62415dc60 0x7fe62406cc50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 -- 172.21.15.129:0/4008699907 >> 172.21.15.129:0/4008699907 conn(0x7fe624071300 msgr2=0x7fe6240716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 -- 172.21.15.129:0/4008699907 shutdown_connections 2024-03-23T15:54:55.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.776+0000 7fe62a53a640 1 -- 172.21.15.129:0/4008699907 wait complete. 2024-03-23T15:54:55.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.777+0000 7fe62a53a640 1 Processor -- start 2024-03-23T15:54:55.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 -- start start 2024-03-23T15:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62406f570 0x7fe6241f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe62415dc60 0x7fe6241f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe62415e040 0x7fe6241f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe6240b2510 con 0x7fe62415dc60 2024-03-23T15:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6241f3750 con 0x7fe62415e040 2024-03-23T15:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.778+0000 7fe62a53a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe6241f3a30 con 0x7fe62406f570 2024-03-23T15:54:55.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.779+0000 7fe623fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62406f570 0x7fe6241f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:55.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.779+0000 7fe628ad7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe62415e040 0x7fe6241f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:55.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.779+0000 7fe6237fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe62415dc60 0x7fe6241f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:55.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.779+0000 7fe628ad7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe62415e040 0x7fe6241f1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:35880/0 (socket says 172.21.15.129:35880) 2024-03-23T15:54:55.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.779+0000 7fe628ad7640 1 -- 172.21.15.129:0/1165082631 learned_addr learned my addr 172.21.15.129:0/1165082631 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:55.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.784+0000 7fe623fff640 1 -- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe62415e040 msgr2=0x7fe6241f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:55.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.784+0000 7fe623fff640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe62415e040 0x7fe6241f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:55.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.784+0000 7fe623fff640 1 -- 172.21.15.129:0/1165082631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe62415dc60 msgr2=0x7fe6241f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:55.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.784+0000 7fe623fff640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe62415dc60 0x7fe6241f0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:55.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.784+0000 7fe623fff640 1 -- 172.21.15.129:0/1165082631 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe614009400 con 0x7fe62406f570 2024-03-23T15:54:55.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.784+0000 7fe6237fe640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe62415dc60 0x7fe6241f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:54:55.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.788+0000 7fe623fff640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62406f570 0x7fe6241f2e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe614033000 tx=0x7fe6140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:55.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.792+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614032840 con 0x7fe62406f570 2024-03-23T15:54:55.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.792+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6140329c0 con 0x7fe62406f570 2024-03-23T15:54:55.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.792+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6240b1b70 con 0x7fe62406f570 2024-03-23T15:54:55.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.792+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614032d70 con 0x7fe62406f570 2024-03-23T15:54:55.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.792+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6240b2130 con 0x7fe62406f570 2024-03-23T15:54:55.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:55.793+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624075740 con 0x7fe62406f570 2024-03-23T15:54:56.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.049+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe61404e430 con 0x7fe62406f570 2024-03-23T15:54:56.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.049+0000 7fe620ff9640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe61006b840 0x7fe61006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:56.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.049+0000 7fe6237fe640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe61006b840 0x7fe61006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:56.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.050+0000 7fe6237fe640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe61006b840 0x7fe61006dd00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe6240b2e60 tx=0x7fe618009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:56.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.052+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe61407acc0 con 0x7fe62406f570 2024-03-23T15:54:56.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.072+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe614076f60 con 0x7fe62406f570 2024-03-23T15:54:56.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.291+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe62406e450 con 0x7fe62406f570 2024-03-23T15:54:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.307+0000 7fe620ff9640 1 -- 172.21.15.129:0/1165082631 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe61407a620 con 0x7fe62406f570 2024-03-23T15:54:56.310 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:56.310 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe61006b840 msgr2=0x7fe61006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe61006b840 0x7fe61006dd00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe6240b2e60 tx=0x7fe618009190 comp rx=0 tx=0).stop 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62406f570 msgr2=0x7fe6241f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62406f570 0x7fe6241f2e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe614033000 tx=0x7fe6140046c0 comp rx=0 tx=0).stop 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 shutdown_connections 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe61006b840 0x7fe61006dd00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe62415e040 0x7fe6241f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe62415dc60 0x7fe6241f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 --2- 172.21.15.129:0/1165082631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe62406f570 0x7fe6241f2e90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 >> 172.21.15.129:0/1165082631 conn(0x7fe624071300 msgr2=0x7fe6240c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 shutdown_connections 2024-03-23T15:54:56.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:56.310+0000 7fe62a53a640 1 -- 172.21.15.129:0/1165082631 wait complete. 2024-03-23T15:54:56.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:56.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:54:57.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:54:58.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.029+0000 7fd4fb96c640 1 Processor -- start 2024-03-23T15:54:58.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.029+0000 7fd4fb96c640 1 -- start start 2024-03-23T15:54:58.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.030+0000 7fd4fb96c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4f406c410 0x7fd4f415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:58.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.030+0000 7fd4fb96c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:58.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.030+0000 7fd4fb96c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f406c830 con 0x7fd4f406c030 2024-03-23T15:54:58.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.030+0000 7fd4fb96c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd4f40cc370 con 0x7fd4f406c410 2024-03-23T15:54:58.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.030+0000 7fd4fb96c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f406cae0 con 0x7fd4f415de40 2024-03-23T15:54:58.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.031+0000 7fd4f9f09640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:58.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.031+0000 7fd4f8f07640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4f406c410 0x7fd4f415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:58.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.031+0000 7fd4f9708640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd4f406c030 0x7fd4f40cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:58596/0 (socket says 172.21.15.129:58596) 2024-03-23T15:54:58.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.031+0000 7fd4f9708640 1 -- 172.21.15.129:0/3974446666 learned_addr learned my addr 172.21.15.129:0/3974446666 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:54:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.034+0000 7fd4f9f09640 1 -- 172.21.15.129:0/3974446666 >> v2:172.21.15.157:3300/0 conn(0x7fd4f406c410 msgr2=0x7fd4f415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:54:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.034+0000 7fd4f9f09640 1 --2- 172.21.15.129:0/3974446666 >> v2:172.21.15.157:3300/0 conn(0x7fd4f406c410 0x7fd4f415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.034+0000 7fd4f9f09640 1 -- 172.21.15.129:0/3974446666 >> v1:172.21.15.157:6789/0 conn(0x7fd4f406c030 legacy=0x7fd4f40cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.034+0000 7fd4f9f09640 1 -- 172.21.15.129:0/3974446666 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f40cabc0 con 0x7fd4f415de40 2024-03-23T15:54:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.037+0000 7fd4f9f09640 1 --2- 172.21.15.129:0/3974446666 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f415e2b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009d90 tx=0x7fd4ec02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=334f7db1824ce1bd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:58.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.039+0000 7fd4ebfff640 1 -- 172.21.15.129:0/3974446666 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec02cc50 con 0x7fd4f415de40 2024-03-23T15:54:58.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4ebfff640 1 -- 172.21.15.129:0/3974446666 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ec02cdb0 con 0x7fd4f415de40 2024-03-23T15:54:58.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4ebfff640 1 -- 172.21.15.129:0/3974446666 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec034660 con 0x7fd4f415de40 2024-03-23T15:54:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 -- 172.21.15.129:0/3974446666 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 msgr2=0x7fd4f415e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/3974446666 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f415e2b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009d90 tx=0x7fd4ec02bea0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 -- 172.21.15.129:0/3974446666 shutdown_connections 2024-03-23T15:54:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/3974446666 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f415e2b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/3974446666 >> v2:172.21.15.157:3300/0 conn(0x7fd4f406c410 0x7fd4f415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 -- 172.21.15.129:0/3974446666 >> 172.21.15.129:0/3974446666 conn(0x7fd4f40cfb20 msgr2=0x7fd4f40cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:58.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 -- 172.21.15.129:0/3974446666 shutdown_connections 2024-03-23T15:54:58.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.042+0000 7fd4fb96c640 1 -- 172.21.15.129:0/3974446666 wait complete. 2024-03-23T15:54:58.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 Processor -- start 2024-03-23T15:54:58.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 -- start start 2024-03-23T15:54:58.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4f406c030 0x7fd4f40b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:58.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 0x7fd4f40b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:58.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:58.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f4151640 con 0x7fd4f406c410 2024-03-23T15:54:58.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f4150ac0 con 0x7fd4f406c030 2024-03-23T15:54:58.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.044+0000 7fd4fb96c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f41f0bd0 con 0x7fd4f415de40 2024-03-23T15:54:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.045+0000 7fd4f8f07640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 0x7fd4f40b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.045+0000 7fd4f9f09640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.045+0000 7fd4f9708640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4f406c030 0x7fd4f40b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:58.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.045+0000 7fd4f8f07640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 0x7fd4f40b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33438/0 (socket says 172.21.15.129:33438) 2024-03-23T15:54:58.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.045+0000 7fd4f8f07640 1 -- 172.21.15.129:0/2884263064 learned_addr learned my addr 172.21.15.129:0/2884263064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:54:58.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.049+0000 7fd4f8f07640 1 -- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 msgr2=0x7fd4f414dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:58.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.049+0000 7fd4f8f07640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f414dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.049+0000 7fd4f8f07640 1 -- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4f406c030 msgr2=0x7fd4f40b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:58.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.049+0000 7fd4f8f07640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4f406c030 0x7fd4f40b2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.049+0000 7fd4f8f07640 1 -- 172.21.15.129:0/2884263064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ec009a70 con 0x7fd4f406c410 2024-03-23T15:54:58.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.051+0000 7fd4f8f07640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 0x7fd4f40b2e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400e9b0 tx=0x7fd4e400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:58.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.055+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4009670 con 0x7fd4f406c410 2024-03-23T15:54:58.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.055+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4e400c9a0 con 0x7fd4f406c410 2024-03-23T15:54:58.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.055+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e400cd50 con 0x7fd4f406c410 2024-03-23T15:54:58.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.055+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f41efd00 con 0x7fd4f406c410 2024-03-23T15:54:58.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.055+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f41f02a0 con 0x7fd4f406c410 2024-03-23T15:54:58.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.056+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f4075740 con 0x7fd4f406c410 2024-03-23T15:54:58.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.308+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd4e4013470 con 0x7fd4f406c410 2024-03-23T15:54:58.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.309+0000 7fd4e9ffb640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd4d806b810 0x7fd4d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:54:58.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.309+0000 7fd4f9708640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd4d806b810 0x7fd4d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:54:58.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.310+0000 7fd4f9708640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd4d806b810 0x7fd4d806dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0007fd0 tx=0x7fd4f0007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:54:58.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.322+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd4e4014e40 con 0x7fd4f406c410 2024-03-23T15:54:58.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.322+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4e4055610 con 0x7fd4f406c410 2024-03-23T15:54:58.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.541+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4f40d0a30 con 0x7fd4f406c410 2024-03-23T15:54:58.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.598+0000 7fd4e9ffb640 1 -- 172.21.15.129:0/2884263064 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd4e4021090 con 0x7fd4f406c410 2024-03-23T15:54:58.601 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:54:58.601 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:54:58.601 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:54:58.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd4d806b810 msgr2=0x7fd4d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:58.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd4d806b810 0x7fd4d806dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0007fd0 tx=0x7fd4f0007e80 comp rx=0 tx=0).stop 2024-03-23T15:54:58.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 msgr2=0x7fd4f40b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:54:58.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 0x7fd4f40b2e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400e9b0 tx=0x7fd4e400ee70 comp rx=0 tx=0).stop 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 shutdown_connections 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4f415de40 0x7fd4f414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f406c410 0x7fd4f40b2e80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd4d806b810 0x7fd4d806dcd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 --2- 172.21.15.129:0/2884263064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4f406c030 0x7fd4f40b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 >> 172.21.15.129:0/2884263064 conn(0x7fd4f40cfb20 msgr2=0x7fd4f40c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 shutdown_connections 2024-03-23T15:54:58.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:54:58.600+0000 7fd4fb96c640 1 -- 172.21.15.129:0/2884263064 wait complete. 2024-03-23T15:54:59.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:54:59.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:00.221 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:00.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.319+0000 7f21db288640 1 Processor -- start 2024-03-23T15:55:00.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.320+0000 7f21db288640 1 -- start start 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.321+0000 7f21db288640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 0x7f21d406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.321+0000 7f21db288640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f21d40cc080 0x7f21d415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.321+0000 7f21db288640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21d406cae0 con 0x7f21d40cbc60 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.321+0000 7f21db288640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f21d40cabc0 con 0x7f21d40cc080 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.321+0000 7f21db288640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f21d40caea0 con 0x7f21d406c030 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.321+0000 7f21d9825640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f21d40cc080 0x7f21d415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.322+0000 7f21d9024640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 0x7f21d406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.322+0000 7f21d9024640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 0x7f21d406c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:45852/0 (socket says 172.21.15.129:45852) 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.322+0000 7f21d9024640 1 -- 172.21.15.129:0/2678100642 learned_addr learned my addr 172.21.15.129:0/2678100642 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.324+0000 7f21d9024640 1 -- 172.21.15.129:0/2678100642 >> v2:172.21.15.129:3300/0 conn(0x7f21d40cc080 msgr2=0x7f21d415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.325+0000 7f21d9024640 1 --2- 172.21.15.129:0/2678100642 >> v2:172.21.15.129:3300/0 conn(0x7f21d40cc080 0x7f21d415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.325+0000 7f21d9024640 1 -- 172.21.15.129:0/2678100642 >> v1:172.21.15.129:6789/0 conn(0x7f21d40cbc60 legacy=0x7f21d415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.325+0000 7f21d9024640 1 -- 172.21.15.129:0/2678100642 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d40cb180 con 0x7f21d406c030 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.327+0000 7f21d9024640 1 --2- 172.21.15.129:0/2678100642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 0x7f21d406c410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009870 tx=0x7f21d002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb33176941e4a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.331+0000 7f21cbfff640 1 -- 172.21.15.129:0/2678100642 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0031640 con 0x7f21d406c030 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.331+0000 7f21cbfff640 1 -- 172.21.15.129:0/2678100642 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21d00317a0 con 0x7f21d406c030 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21cbfff640 1 -- 172.21.15.129:0/2678100642 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0031b30 con 0x7f21d406c030 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 -- 172.21.15.129:0/2678100642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 msgr2=0x7f21d406c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 --2- 172.21.15.129:0/2678100642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 0x7f21d406c410 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009870 tx=0x7f21d002b8f0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 -- 172.21.15.129:0/2678100642 shutdown_connections 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 --2- 172.21.15.129:0/2678100642 >> v2:172.21.15.129:3300/0 conn(0x7f21d40cc080 0x7f21d415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 --2- 172.21.15.129:0/2678100642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d406c030 0x7f21d406c410 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 -- 172.21.15.129:0/2678100642 >> 172.21.15.129:0/2678100642 conn(0x7f21d40cfb20 msgr2=0x7f21d40cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.332+0000 7f21db288640 1 -- 172.21.15.129:0/2678100642 shutdown_connections 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.333+0000 7f21db288640 1 -- 172.21.15.129:0/2678100642 wait complete. 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.334+0000 7f21db288640 1 Processor -- start 2024-03-23T15:55:00.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.334+0000 7f21db288640 1 -- start start 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.334+0000 7f21db288640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 0x7f21d41572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.335+0000 7f21db288640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d40cbc60 0x7f21d41f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.335+0000 7f21db288640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d40cc080 0x7f21d414e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.335+0000 7f21db288640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21d415c3e0 con 0x7f21d406c030 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.335+0000 7f21db288640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f21d41e5820 con 0x7f21d40cbc60 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.335+0000 7f21db288640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f21d40c40a0 con 0x7f21d40cc080 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.335+0000 7f21d9024640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 0x7f21d41572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.336+0000 7f21d9024640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 0x7f21d41572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33454/0 (socket says 172.21.15.129:33454) 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.336+0000 7f21d9024640 1 -- 172.21.15.129:0/1762495981 learned_addr learned my addr 172.21.15.129:0/1762495981 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.336+0000 7f21d9825640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d40cc080 0x7f21d414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.338+0000 7f21d8823640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d40cbc60 0x7f21d41f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.339+0000 7f21d9024640 1 -- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d40cc080 msgr2=0x7f21d414e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.339+0000 7f21d9024640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d40cc080 0x7f21d414e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.339+0000 7f21d9024640 1 -- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d40cbc60 msgr2=0x7f21d41f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.339+0000 7f21d9024640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d40cbc60 0x7f21d41f4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.339+0000 7f21d9024640 1 -- 172.21.15.129:0/1762495981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d00094d0 con 0x7f21d406c030 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.342+0000 7f21d9024640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 0x7f21d41572c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f21d0000c00 tx=0x7f21d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.345+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d003a930 con 0x7f21d406c030 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.345+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d415beb0 con 0x7f21d406c030 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.345+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d41f3e00 con 0x7f21d406c030 2024-03-23T15:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.346+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d406b8a0 con 0x7f21d406c030 2024-03-23T15:55:00.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.603+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21d003aa90 con 0x7f21d406c030 2024-03-23T15:55:00.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.603+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d003ae00 con 0x7f21d406c030 2024-03-23T15:55:00.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.604+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f21d002f050 con 0x7f21d406c030 2024-03-23T15:55:00.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.605+0000 7f21c9ffb640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f21b806b840 0x7f21b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:00.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.605+0000 7f21d8823640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f21b806b840 0x7f21b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:00.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.605+0000 7f21d8823640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f21b806b840 0x7f21b806dd00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f21d41e4810 tx=0x7f21c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:00.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.627+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f21d007aaf0 con 0x7f21d406c030 2024-03-23T15:55:00.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.627+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f21d0076d90 con 0x7f21d406c030 2024-03-23T15:55:00.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.856+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21d40d0640 con 0x7f21d406c030 2024-03-23T15:55:00.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.871+0000 7f21c9ffb640 1 -- 172.21.15.129:0/1762495981 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f21d007a450 con 0x7f21d406c030 2024-03-23T15:55:00.874 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:00.874 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:00.875 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:00.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f21b806b840 msgr2=0x7f21b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:00.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f21b806b840 0x7f21b806dd00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f21d41e4810 tx=0x7f21c4009040 comp rx=0 tx=0).stop 2024-03-23T15:55:00.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 msgr2=0x7f21d41572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 0x7f21d41572c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f21d0000c00 tx=0x7f21d00046c0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 shutdown_connections 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21d40cc080 0x7f21d414e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f21b806b840 0x7f21b806dd00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21d40cbc60 0x7f21d41f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 --2- 172.21.15.129:0/1762495981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21d406c030 0x7f21d41572c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 >> 172.21.15.129:0/1762495981 conn(0x7f21d40cfb20 msgr2=0x7f21d40b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 shutdown_connections 2024-03-23T15:55:00.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:00.873+0000 7f21db288640 1 -- 172.21.15.129:0/1762495981 wait complete. 2024-03-23T15:55:01.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:01.496 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:02.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.596+0000 7f4663325640 1 Processor -- start 2024-03-23T15:55:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.596+0000 7f4663325640 1 -- start start 2024-03-23T15:55:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.598+0000 7f4663325640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f465c15c000 0x7f465c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.598+0000 7f4663325640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465c15b290 con 0x7f465c0ceb00 2024-03-23T15:55:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.598+0000 7f4663325640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465c0ce800 con 0x7f465c0ccd60 2024-03-23T15:55:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.598+0000 7f4663325640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f465c154a70 con 0x7f465c15c000 2024-03-23T15:55:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.598+0000 7f46608c0640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f465c0ceb00 0x7f465c15b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59508/0 (socket says 172.21.15.129:59508) 2024-03-23T15:55:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.598+0000 7f46608c0640 1 -- 172.21.15.129:0/66411425 learned_addr learned my addr 172.21.15.129:0/66411425 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.599+0000 7f46618c2640 1 --2- 172.21.15.129:0/66411425 >> v2:172.21.15.157:3300/0 conn(0x7f465c15c000 0x7f465c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.601+0000 7f4653fff640 1 -- 172.21.15.129:0/66411425 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574815820 0 0) 0x7f464c001110 con 0x7f465c0ceb00 2024-03-23T15:55:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.602+0000 7f4653fff640 1 -- 172.21.15.129:0/66411425 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4648003520 con 0x7f465c0ceb00 2024-03-23T15:55:02.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.603+0000 7f46618c2640 1 -- 172.21.15.129:0/66411425 >> v1:172.21.15.157:6789/0 conn(0x7f465c0ccd60 legacy=0x7f465c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:02.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.603+0000 7f46618c2640 1 -- 172.21.15.129:0/66411425 >> v1:172.21.15.129:6789/0 conn(0x7f465c0ceb00 legacy=0x7f465c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:02.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.603+0000 7f46618c2640 1 -- 172.21.15.129:0/66411425 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c0cd950 con 0x7f465c15c000 2024-03-23T15:55:02.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.609+0000 7f46618c2640 1 --2- 172.21.15.129:0/66411425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c15ad50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4654009d90 tx=0x7f465402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cd29eaa9c164e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:02.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.613+0000 7f4653fff640 1 -- 172.21.15.129:0/66411425 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465402cd80 con 0x7f465c15c000 2024-03-23T15:55:02.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.613+0000 7f4653fff640 1 -- 172.21.15.129:0/66411425 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4654034440 con 0x7f465c15c000 2024-03-23T15:55:02.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.613+0000 7f4653fff640 1 -- 172.21.15.129:0/66411425 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46540347d0 con 0x7f465c15c000 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 -- 172.21.15.129:0/66411425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 msgr2=0x7f465c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 --2- 172.21.15.129:0/66411425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c15ad50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4654009d90 tx=0x7f465402bfd0 comp rx=0 tx=0).stop 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 -- 172.21.15.129:0/66411425 shutdown_connections 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 --2- 172.21.15.129:0/66411425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c15ad50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 -- 172.21.15.129:0/66411425 >> 172.21.15.129:0/66411425 conn(0x7f465c0b1840 msgr2=0x7f465c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 -- 172.21.15.129:0/66411425 shutdown_connections 2024-03-23T15:55:02.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.614+0000 7f4663325640 1 -- 172.21.15.129:0/66411425 wait complete. 2024-03-23T15:55:02.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.615+0000 7f4663325640 1 Processor -- start 2024-03-23T15:55:02.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.615+0000 7f4663325640 1 -- start start 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f4663325640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 0x7f465c1f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f4663325640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f465c0ceb00 0x7f465c158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f4663325640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f4663325640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f465c1f45e0 con 0x7f465c0ccd60 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f4663325640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f465c1f4320 con 0x7f465c15c000 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f4663325640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f465c0cb220 con 0x7f465c0ceb00 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f46610c1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 0x7f465c1f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:02.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.616+0000 7f46618c2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:02.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.617+0000 7f46608c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f465c0ceb00 0x7f465c158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:02.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.617+0000 7f46610c1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 0x7f465c1f39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59014/0 (socket says 172.21.15.129:59014) 2024-03-23T15:55:02.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.617+0000 7f46610c1640 1 -- 172.21.15.129:0/776717139 learned_addr learned my addr 172.21.15.129:0/776717139 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:02.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.620+0000 7f46610c1640 1 -- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f465c0ceb00 msgr2=0x7f465c158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:02.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.620+0000 7f46610c1640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f465c0ceb00 0x7f465c158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:02.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.620+0000 7f46610c1640 1 -- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 msgr2=0x7f465c158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:02.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.620+0000 7f46610c1640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:02.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.620+0000 7f46610c1640 1 -- 172.21.15.129:0/776717139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4654009a70 con 0x7f465c0ccd60 2024-03-23T15:55:02.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.622+0000 7f46610c1640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 0x7f465c1f39f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f465800c400 tx=0x7f465800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:02.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.627+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46580052d0 con 0x7f465c0ccd60 2024-03-23T15:55:02.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.627+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4658005430 con 0x7f465c0ccd60 2024-03-23T15:55:02.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.627+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658003830 con 0x7f465c0ccd60 2024-03-23T15:55:02.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.627+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465c0cb500 con 0x7f465c0ccd60 2024-03-23T15:55:02.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.627+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f465c157f30 con 0x7f465c0ccd60 2024-03-23T15:55:02.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.628+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465c04f530 con 0x7f465c0ccd60 2024-03-23T15:55:02.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.651+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4658003990 con 0x7f465c0ccd60 2024-03-23T15:55:02.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.651+0000 7f4651ffb640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f464006b810 0x7f464006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:02.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.652+0000 7f46608c0640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f464006b810 0x7f464006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:02.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.652+0000 7f46608c0640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f464006b810 0x7f464006dcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f465c153eb0 tx=0x7f464c007bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:02.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.918+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4658058360 con 0x7f465c0ccd60 2024-03-23T15:55:02.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:02.918+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4658054600 con 0x7f465c0ccd60 2024-03-23T15:55:03.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.162+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f465c15a120 con 0x7f465c0ccd60 2024-03-23T15:55:03.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.208+0000 7f4651ffb640 1 -- 172.21.15.129:0/776717139 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f465c15a120 con 0x7f465c0ccd60 2024-03-23T15:55:03.213 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:03.213 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:03.214 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:03.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f464006b810 msgr2=0x7f464006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:03.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f464006b810 0x7f464006dcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f465c153eb0 tx=0x7f464c007bb0 comp rx=0 tx=0).stop 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 msgr2=0x7f465c1f39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 0x7f465c1f39f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f465800c400 tx=0x7f465800c8c0 comp rx=0 tx=0).stop 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 shutdown_connections 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f464006b810 0x7f464006dcd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f465c15c000 0x7f465c158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f465c0ceb00 0x7f465c158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 --2- 172.21.15.129:0/776717139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f465c0ccd60 0x7f465c1f39f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 >> 172.21.15.129:0/776717139 conn(0x7f465c0b1840 msgr2=0x7f465c15ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 shutdown_connections 2024-03-23T15:55:03.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:03.211+0000 7f4663325640 1 -- 172.21.15.129:0/776717139 wait complete. 2024-03-23T15:55:03.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:03.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:04.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:04.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.929+0000 7f1647cd4640 1 Processor -- start 2024-03-23T15:55:04.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.929+0000 7f1647cd4640 1 -- start start 2024-03-23T15:55:04.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.930+0000 7f1647cd4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f164015dba0 0x7f164006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:04.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f1647cd4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f16400cca30 0x7f164006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:04.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f1647cd4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16400c4c00 con 0x7f164015df80 2024-03-23T15:55:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f1647cd4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f16400ccea0 con 0x7f164015dba0 2024-03-23T15:55:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f1647cd4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f164006c240 con 0x7f16400cca30 2024-03-23T15:55:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f164526f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f16400cca30 0x7f164006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f1645a70640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f164015dba0 0x7f164006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f164526f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f16400cca30 0x7f164006ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44964/0 (socket says 172.21.15.129:44964) 2024-03-23T15:55:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.931+0000 7f164526f640 1 -- 172.21.15.129:0/2335844133 learned_addr learned my addr 172.21.15.129:0/2335844133 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:04.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.934+0000 7f164526f640 1 -- 172.21.15.129:0/2335844133 >> v2:172.21.15.129:3300/0 conn(0x7f164015dba0 msgr2=0x7f164006cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:04.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.934+0000 7f164526f640 1 --2- 172.21.15.129:0/2335844133 >> v2:172.21.15.129:3300/0 conn(0x7f164015dba0 0x7f164006cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:04.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.934+0000 7f164526f640 1 -- 172.21.15.129:0/2335844133 >> v1:172.21.15.129:6789/0 conn(0x7f164015df80 legacy=0x7f16401540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:04.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.934+0000 7f164526f640 1 -- 172.21.15.129:0/2335844133 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16400cbb20 con 0x7f16400cca30 2024-03-23T15:55:04.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.936+0000 7f164526f640 1 --2- 172.21.15.129:0/2335844133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16400cca30 0x7f164006ff30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1630009850 tx=0x7f163002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e24260d8dec2824d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.967+0000 7f1644a6e640 1 -- 172.21.15.129:0/2335844133 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630004230 con 0x7f16400cca30 2024-03-23T15:55:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.975+0000 7f1644a6e640 1 -- 172.21.15.129:0/2335844133 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1630004390 con 0x7f16400cca30 2024-03-23T15:55:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.975+0000 7f1644a6e640 1 -- 172.21.15.129:0/2335844133 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630030840 con 0x7f16400cca30 2024-03-23T15:55:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 -- 172.21.15.129:0/2335844133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16400cca30 msgr2=0x7f164006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 --2- 172.21.15.129:0/2335844133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16400cca30 0x7f164006ff30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1630009850 tx=0x7f163002b980 comp rx=0 tx=0).stop 2024-03-23T15:55:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 -- 172.21.15.129:0/2335844133 shutdown_connections 2024-03-23T15:55:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 --2- 172.21.15.129:0/2335844133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f16400cca30 0x7f164006ff30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 --2- 172.21.15.129:0/2335844133 >> v2:172.21.15.129:3300/0 conn(0x7f164015dba0 0x7f164006cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:04.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 -- 172.21.15.129:0/2335844133 >> 172.21.15.129:0/2335844133 conn(0x7f1640071300 msgr2=0x7f16400716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:04.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 -- 172.21.15.129:0/2335844133 shutdown_connections 2024-03-23T15:55:04.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.976+0000 7f1647cd4640 1 -- 172.21.15.129:0/2335844133 wait complete. 2024-03-23T15:55:04.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.978+0000 7f1647cd4640 1 Processor -- start 2024-03-23T15:55:04.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.978+0000 7f1647cd4640 1 -- start start 2024-03-23T15:55:04.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1647cd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16400cca30 0x7f16400c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1647cd4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 0x7f1640159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1647cd4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f164015df80 0x7f1640150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1647cd4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1640157390 con 0x7f164015df80 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1647cd4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f16400c5d30 con 0x7f164015dba0 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1647cd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f16401e36c0 con 0x7f16400cca30 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f164526f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 0x7f1640159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f164526f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 0x7f1640159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44968/0 (socket says 172.21.15.129:44968) 2024-03-23T15:55:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f164526f640 1 -- 172.21.15.129:0/909686517 learned_addr learned my addr 172.21.15.129:0/909686517 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.980+0000 7f1645a70640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16400cca30 0x7f16400c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.984+0000 7f164526f640 1 -- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16400cca30 msgr2=0x7f16400c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.984+0000 7f164526f640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16400cca30 0x7f16400c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.984+0000 7f164526f640 1 -- 172.21.15.129:0/909686517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f164015df80 msgr2=0x7f1640150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.984+0000 7f164526f640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f164015df80 0x7f1640150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.984+0000 7f164526f640 1 -- 172.21.15.129:0/909686517 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1630009460 con 0x7f164015dba0 2024-03-23T15:55:04.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.986+0000 7f164526f640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 0x7f1640159900 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1630009970 tx=0x7f16300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:04.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.990+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163003e070 con 0x7f164015dba0 2024-03-23T15:55:04.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.990+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1630004500 con 0x7f164015dba0 2024-03-23T15:55:04.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.990+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16400c4da0 con 0x7f164015dba0 2024-03-23T15:55:04.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.990+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630030dc0 con 0x7f164015dba0 2024-03-23T15:55:04.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.990+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16400c5360 con 0x7f164015dba0 2024-03-23T15:55:04.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:04.991+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16400686f0 con 0x7f164015dba0 2024-03-23T15:55:05.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.252+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1630003610 con 0x7f164015dba0 2024-03-23T15:55:05.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.252+0000 7f16367fc640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f162406b860 0x7f162406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:05.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.253+0000 7f1645a70640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f162406b860 0x7f162406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:05.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.253+0000 7f1645a70640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f162406b860 0x7f162406dd20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f163c009b40 tx=0x7f163c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:05.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.262+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f163007afa0 con 0x7f164015dba0 2024-03-23T15:55:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.273+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1630077240 con 0x7f164015dba0 2024-03-23T15:55:05.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.524+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16400c4be0 con 0x7f164015dba0 2024-03-23T15:55:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.538+0000 7f16367fc640 1 -- 172.21.15.129:0/909686517 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1630048090 con 0x7f164015dba0 2024-03-23T15:55:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:05.542 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:05.542 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f162406b860 msgr2=0x7f162406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f162406b860 0x7f162406dd20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f163c009b40 tx=0x7f163c009190 comp rx=0 tx=0).stop 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 msgr2=0x7f1640159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 0x7f1640159900 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1630009970 tx=0x7f16300046c0 comp rx=0 tx=0).stop 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 shutdown_connections 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f164015df80 0x7f1640150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f162406b860 0x7f162406dd20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f164015dba0 0x7f1640159900 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 --2- 172.21.15.129:0/909686517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16400cca30 0x7f16400c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 >> 172.21.15.129:0/909686517 conn(0x7f1640071300 msgr2=0x7f164015b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 shutdown_connections 2024-03-23T15:55:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:05.541+0000 7f1647cd4640 1 -- 172.21.15.129:0/909686517 wait complete. 2024-03-23T15:55:06.159 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:06.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:07.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:07.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.261+0000 7f2fe9be4640 1 Processor -- start 2024-03-23T15:55:07.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.261+0000 7f2fe9be4640 1 -- start start 2024-03-23T15:55:07.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.262+0000 7f2fe9be4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2fe41544b0 0x7f2fe415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:07.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.262+0000 7f2fe9be4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2fe40c6000 0x7f2fe40c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:07.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.262+0000 7f2fe9be4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe40cbcb0 con 0x7f2fe40caa00 2024-03-23T15:55:07.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.262+0000 7f2fe9be4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f2fe40c5be0 con 0x7f2fe41544b0 2024-03-23T15:55:07.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.262+0000 7f2fe9be4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2fe40c69b0 con 0x7f2fe40c6000 2024-03-23T15:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.262+0000 7f2fe37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2fe41544b0 0x7f2fe415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.263+0000 7f2fe3fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2fe40c6000 0x7f2fe40c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.263+0000 7f2fe37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2fe41544b0 0x7f2fe415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59052/0 (socket says 172.21.15.129:59052) 2024-03-23T15:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.263+0000 7f2fe37fe640 1 -- 172.21.15.129:0/3473294461 learned_addr learned my addr 172.21.15.129:0/3473294461 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.265+0000 7f2fe37fe640 1 -- 172.21.15.129:0/3473294461 >> v2:172.21.15.157:3300/0 conn(0x7f2fe40c6000 msgr2=0x7f2fe40c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.265+0000 7f2fe37fe640 1 --2- 172.21.15.129:0/3473294461 >> v2:172.21.15.157:3300/0 conn(0x7f2fe40c6000 0x7f2fe40c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.265+0000 7f2fe37fe640 1 -- 172.21.15.129:0/3473294461 >> v1:172.21.15.157:6789/0 conn(0x7f2fe40caa00 legacy=0x7f2fe406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.265+0000 7f2fe37fe640 1 -- 172.21.15.129:0/3473294461 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe415b280 con 0x7f2fe41544b0 2024-03-23T15:55:07.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.267+0000 7f2fe37fe640 1 --2- 172.21.15.129:0/3473294461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe41544b0 0x7f2fe415a2e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40b3b50 tx=0x7f2fd402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36981a0d9d7a0c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.270+0000 7f2fe27fc640 1 -- 172.21.15.129:0/3473294461 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd402f850 con 0x7f2fe41544b0 2024-03-23T15:55:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.272+0000 7f2fe27fc640 1 -- 172.21.15.129:0/3473294461 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd402f9b0 con 0x7f2fe41544b0 2024-03-23T15:55:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.272+0000 7f2fe27fc640 1 -- 172.21.15.129:0/3473294461 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd402fd40 con 0x7f2fe41544b0 2024-03-23T15:55:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3473294461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe41544b0 msgr2=0x7f2fe415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3473294461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe41544b0 0x7f2fe415a2e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40b3b50 tx=0x7f2fd402ba30 comp rx=0 tx=0).stop 2024-03-23T15:55:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3473294461 shutdown_connections 2024-03-23T15:55:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3473294461 >> v2:172.21.15.157:3300/0 conn(0x7f2fe40c6000 0x7f2fe40c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3473294461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe41544b0 0x7f2fe415a2e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3473294461 >> 172.21.15.129:0/3473294461 conn(0x7f2fe4061b50 msgr2=0x7f2fe4061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:07.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3473294461 shutdown_connections 2024-03-23T15:55:07.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.273+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3473294461 wait complete. 2024-03-23T15:55:07.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.274+0000 7f2fe9be4640 1 Processor -- start 2024-03-23T15:55:07.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.274+0000 7f2fe9be4640 1 -- start start 2024-03-23T15:55:07.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe9be4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c6000 0x7f2fe41f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:07.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe9be4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 0x7f2fe41f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe9be4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe41544b0 0x7f2fe40c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe9be4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe4154b30 con 0x7f2fe40caa00 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe9be4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe40c8c30 con 0x7f2fe41544b0 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe9be4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe41f3c80 con 0x7f2fe40c6000 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.275+0000 7f2fe2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 0x7f2fe41f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.276+0000 7f2fe3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe41544b0 0x7f2fe40c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.276+0000 7f2fe37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c6000 0x7f2fe41f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.276+0000 7f2fe2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 0x7f2fe41f3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59060/0 (socket says 172.21.15.129:59060) 2024-03-23T15:55:07.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.276+0000 7f2fe2ffd640 1 -- 172.21.15.129:0/3658570750 learned_addr learned my addr 172.21.15.129:0/3658570750 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:07.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.281+0000 7f2fe2ffd640 1 -- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c6000 msgr2=0x7f2fe41f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.281+0000 7f2fe2ffd640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c6000 0x7f2fe41f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.281+0000 7f2fe2ffd640 1 -- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe41544b0 msgr2=0x7f2fe40c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.281+0000 7f2fe2ffd640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe41544b0 0x7f2fe40c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.281+0000 7f2fe2ffd640 1 -- 172.21.15.129:0/3658570750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd40094d0 con 0x7f2fe40caa00 2024-03-23T15:55:07.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.284+0000 7f2fe2ffd640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 0x7f2fe41f3660 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00c400 tx=0x7f2fcc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:07.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.290+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0052d0 con 0x7f2fe40caa00 2024-03-23T15:55:07.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.290+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fcc005450 con 0x7f2fe40caa00 2024-03-23T15:55:07.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.290+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc007260 con 0x7f2fe40caa00 2024-03-23T15:55:07.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.290+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe4157470 con 0x7f2fe40caa00 2024-03-23T15:55:07.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.290+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe41579f0 con 0x7f2fe40caa00 2024-03-23T15:55:07.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.292+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe4075740 con 0x7f2fe40caa00 2024-03-23T15:55:07.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.321+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2fcc0073c0 con 0x7f2fe40caa00 2024-03-23T15:55:07.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.322+0000 7f2fdbfff640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2fc406b860 0x7f2fc406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:07.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.322+0000 7f2fe37fe640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2fc406b860 0x7f2fc406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:07.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.322+0000 7f2fe37fe640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2fc406b860 0x7f2fc406dd20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4036000 tx=0x7f2fd4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:07.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.327+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2fcc007670 con 0x7f2fe40caa00 2024-03-23T15:55:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.576+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2fcc054890 con 0x7f2fe40caa00 2024-03-23T15:55:07.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:55:07.817 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:55:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.845+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fe4157d00 con 0x7f2fe40caa00 2024-03-23T15:55:07.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.865+0000 7f2fdbfff640 1 -- 172.21.15.129:0/3658570750 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2fcc020020 con 0x7f2fe40caa00 2024-03-23T15:55:07.869 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:07.869 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:07.869 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2fc406b860 msgr2=0x7f2fc406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2fc406b860 0x7f2fc406dd20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4036000 tx=0x7f2fd4008030 comp rx=0 tx=0).stop 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 msgr2=0x7f2fe41f3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 0x7f2fe41f3660 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00c400 tx=0x7f2fcc00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 shutdown_connections 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2fc406b860 0x7f2fc406dd20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe41544b0 0x7f2fe40c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fe40caa00 0x7f2fe41f3660 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 --2- 172.21.15.129:0/3658570750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c6000 0x7f2fe41f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 >> 172.21.15.129:0/3658570750 conn(0x7f2fe4061b50 msgr2=0x7f2fe414a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 shutdown_connections 2024-03-23T15:55:07.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:07.869+0000 7f2fe9be4640 1 -- 172.21.15.129:0/3658570750 wait complete. 2024-03-23T15:55:08.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:08.492 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:09.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:09.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.593+0000 7fdff009d640 1 Processor -- start 2024-03-23T15:55:09.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.593+0000 7fdff009d640 1 -- start start 2024-03-23T15:55:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.594+0000 7fdff009d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80ccd60 0x7fdfe815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.594+0000 7fdff009d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdfe80ceb00 0x7fdfe806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.594+0000 7fdff009d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfe815b290 con 0x7fdfe80cef40 2024-03-23T15:55:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.594+0000 7fdff009d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdfe80ce800 con 0x7fdfe80ceb00 2024-03-23T15:55:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.594+0000 7fdff009d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe8154a70 con 0x7fdfe80ccd60 2024-03-23T15:55:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.595+0000 7fdfede39640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80ccd60 0x7fdfe815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.595+0000 7fdfed638640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdfe80ceb00 0x7fdfe806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.595+0000 7fdfee63a640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fdfe80cef40 0x7fdfe815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:39034/0 (socket says 172.21.15.129:39034) 2024-03-23T15:55:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.595+0000 7fdfee63a640 1 -- 172.21.15.129:0/4046011644 learned_addr learned my addr 172.21.15.129:0/4046011644 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.599+0000 7fdfede39640 1 -- 172.21.15.129:0/4046011644 >> v2:172.21.15.157:3300/0 conn(0x7fdfe80ceb00 msgr2=0x7fdfe806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.599+0000 7fdfede39640 1 --2- 172.21.15.129:0/4046011644 >> v2:172.21.15.157:3300/0 conn(0x7fdfe80ceb00 0x7fdfe806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.599+0000 7fdfede39640 1 -- 172.21.15.129:0/4046011644 >> v1:172.21.15.157:6789/0 conn(0x7fdfe80cef40 legacy=0x7fdfe815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.599+0000 7fdfede39640 1 -- 172.21.15.129:0/4046011644 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe80cd950 con 0x7fdfe80ccd60 2024-03-23T15:55:09.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.602+0000 7fdfede39640 1 --2- 172.21.15.129:0/4046011644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80ccd60 0x7fdfe815ad50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40097a0 tx=0x7fdfe402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ae91caf11366a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:09.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.605+0000 7fdfece37640 1 -- 172.21.15.129:0/4046011644 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4030610 con 0x7fdfe80ccd60 2024-03-23T15:55:09.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.608+0000 7fdfece37640 1 -- 172.21.15.129:0/4046011644 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe4030770 con 0x7fdfe80ccd60 2024-03-23T15:55:09.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.608+0000 7fdfece37640 1 -- 172.21.15.129:0/4046011644 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4030b00 con 0x7fdfe80ccd60 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 -- 172.21.15.129:0/4046011644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80ccd60 msgr2=0x7fdfe815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 --2- 172.21.15.129:0/4046011644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80ccd60 0x7fdfe815ad50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40097a0 tx=0x7fdfe402b960 comp rx=0 tx=0).stop 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 -- 172.21.15.129:0/4046011644 shutdown_connections 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 --2- 172.21.15.129:0/4046011644 >> v2:172.21.15.157:3300/0 conn(0x7fdfe80ceb00 0x7fdfe806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 --2- 172.21.15.129:0/4046011644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80ccd60 0x7fdfe815ad50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 -- 172.21.15.129:0/4046011644 >> 172.21.15.129:0/4046011644 conn(0x7fdfe80b1840 msgr2=0x7fdfe80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 -- 172.21.15.129:0/4046011644 shutdown_connections 2024-03-23T15:55:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.609+0000 7fdff009d640 1 -- 172.21.15.129:0/4046011644 wait complete. 2024-03-23T15:55:09.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.611+0000 7fdff009d640 1 Processor -- start 2024-03-23T15:55:09.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.611+0000 7fdff009d640 1 -- start start 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdff009d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe80ccd60 0x7fdfe8150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdff009d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfe80ceb00 0x7fdfe814f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdff009d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80cef40 0x7fdfe814ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdff009d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe8153da0 con 0x7fdfe80ccd60 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdff009d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe8150ac0 con 0x7fdfe80ceb00 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdff009d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe80b3430 con 0x7fdfe80cef40 2024-03-23T15:55:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdfee63a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80cef40 0x7fdfe814ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:09.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdfede39640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe80ccd60 0x7fdfe8150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:09.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.612+0000 7fdfed638640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfe80ceb00 0x7fdfe814f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:09.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.613+0000 7fdfed638640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfe80ceb00 0x7fdfe814f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44996/0 (socket says 172.21.15.129:44996) 2024-03-23T15:55:09.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.613+0000 7fdfed638640 1 -- 172.21.15.129:0/820292437 learned_addr learned my addr 172.21.15.129:0/820292437 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:09.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.618+0000 7fdfede39640 1 -- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80cef40 msgr2=0x7fdfe814ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:09.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.618+0000 7fdfede39640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80cef40 0x7fdfe814ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:09.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.618+0000 7fdfede39640 1 -- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfe80ceb00 msgr2=0x7fdfe814f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:09.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.618+0000 7fdfede39640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfe80ceb00 0x7fdfe814f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:09.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.618+0000 7fdfede39640 1 -- 172.21.15.129:0/820292437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe4009400 con 0x7fdfe80ccd60 2024-03-23T15:55:09.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.621+0000 7fdfede39640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe80ccd60 0x7fdfe8150580 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4033000 tx=0x7fdfe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:09.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.625+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe403e070 con 0x7fdfe80ccd60 2024-03-23T15:55:09.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.625+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe4030e10 con 0x7fdfe80ccd60 2024-03-23T15:55:09.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.625+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4039c30 con 0x7fdfe80ccd60 2024-03-23T15:55:09.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.625+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe80b36d0 con 0x7fdfe80ccd60 2024-03-23T15:55:09.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.625+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe80b3c90 con 0x7fdfe80ccd60 2024-03-23T15:55:09.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.626+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe804f530 con 0x7fdfe80ccd60 2024-03-23T15:55:09.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.892+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdfe404a3f0 con 0x7fdfe80ccd60 2024-03-23T15:55:09.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.892+0000 7fdfde7fc640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdfcc06b860 0x7fdfcc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:09.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.893+0000 7fdfed638640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdfcc06b860 0x7fdfcc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:09.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.893+0000 7fdfed638640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdfcc06b860 0x7fdfcc06dd20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8153b30 tx=0x7fdfd8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.928+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdfe407a730 con 0x7fdfe80ccd60 2024-03-23T15:55:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:09.928+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdfe40769d0 con 0x7fdfe80ccd60 2024-03-23T15:55:10.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.146+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfe80686f0 con 0x7fdfe80ccd60 2024-03-23T15:55:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.167+0000 7fdfde7fc640 1 -- 172.21.15.129:0/820292437 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fdfe4034020 con 0x7fdfe80ccd60 2024-03-23T15:55:10.170 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:10.170 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:10.170 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdfcc06b860 msgr2=0x7fdfcc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdfcc06b860 0x7fdfcc06dd20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8153b30 tx=0x7fdfd8009190 comp rx=0 tx=0).stop 2024-03-23T15:55:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe80ccd60 msgr2=0x7fdfe8150580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe80ccd60 0x7fdfe8150580 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4033000 tx=0x7fdfe40046c0 comp rx=0 tx=0).stop 2024-03-23T15:55:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 shutdown_connections 2024-03-23T15:55:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfe80cef40 0x7fdfe814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdfcc06b860 0x7fdfcc06dd20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfe80ceb00 0x7fdfe814f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 --2- 172.21.15.129:0/820292437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe80ccd60 0x7fdfe8150580 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 >> 172.21.15.129:0/820292437 conn(0x7fdfe80b1840 msgr2=0x7fdfe8158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 shutdown_connections 2024-03-23T15:55:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:10.169+0000 7fdff009d640 1 -- 172.21.15.129:0/820292437 wait complete. 2024-03-23T15:55:10.787 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:10.788 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:11.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:11.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.888+0000 7fb82acaa640 1 Processor -- start 2024-03-23T15:55:11.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.888+0000 7fb82acaa640 1 -- start start 2024-03-23T15:55:11.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.889+0000 7fb82acaa640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb82415a2e0 0x7fb82406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:11.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.889+0000 7fb82acaa640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8240cbcb0 con 0x7fb8241544b0 2024-03-23T15:55:11.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.889+0000 7fb82acaa640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82415b280 con 0x7fb82406c0b0 2024-03-23T15:55:11.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.889+0000 7fb82acaa640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb8240c5be0 con 0x7fb82415a2e0 2024-03-23T15:55:11.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.890+0000 7fb829247640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb82406c0b0 0x7fb8240c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33908/0 (socket says 172.21.15.129:33908) 2024-03-23T15:55:11.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.890+0000 7fb829247640 1 -- 172.21.15.129:0/3490240920 learned_addr learned my addr 172.21.15.129:0/3490240920 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:11.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.891+0000 7fb823fff640 1 --2- 172.21.15.129:0/3490240920 >> v2:172.21.15.157:3300/0 conn(0x7fb82415a2e0 0x7fb82406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.893+0000 7fb8237fe640 1 -- 172.21.15.129:0/3490240920 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258552841 0 0) 0x7fb8240cbcb0 con 0x7fb8241544b0 2024-03-23T15:55:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.893+0000 7fb8237fe640 1 -- 172.21.15.129:0/3490240920 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb814003520 con 0x7fb8241544b0 2024-03-23T15:55:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.895+0000 7fb823fff640 1 -- 172.21.15.129:0/3490240920 >> v1:172.21.15.157:6789/0 conn(0x7fb82406c0b0 legacy=0x7fb8240c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.895+0000 7fb823fff640 1 -- 172.21.15.129:0/3490240920 >> v1:172.21.15.129:6789/0 conn(0x7fb8241544b0 legacy=0x7fb8240caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.895+0000 7fb823fff640 1 -- 172.21.15.129:0/3490240920 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82415a9e0 con 0x7fb82415a2e0 2024-03-23T15:55:11.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.900+0000 7fb823fff640 1 --2- 172.21.15.129:0/3490240920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb82415a2e0 0x7fb82406bb70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb8100097a0 tx=0x7fb81002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5f8f4799f14332e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.905+0000 7fb8237fe640 1 -- 172.21.15.129:0/3490240920 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81002f810 con 0x7fb82415a2e0 2024-03-23T15:55:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.905+0000 7fb8237fe640 1 -- 172.21.15.129:0/3490240920 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb81002f970 con 0x7fb82415a2e0 2024-03-23T15:55:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.905+0000 7fb8237fe640 1 -- 172.21.15.129:0/3490240920 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81002fd00 con 0x7fb82415a2e0 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 -- 172.21.15.129:0/3490240920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb82415a2e0 msgr2=0x7fb82406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 --2- 172.21.15.129:0/3490240920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb82415a2e0 0x7fb82406bb70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb8100097a0 tx=0x7fb81002b9f0 comp rx=0 tx=0).stop 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 -- 172.21.15.129:0/3490240920 shutdown_connections 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 --2- 172.21.15.129:0/3490240920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb82415a2e0 0x7fb82406bb70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 -- 172.21.15.129:0/3490240920 >> 172.21.15.129:0/3490240920 conn(0x7fb824061b50 msgr2=0x7fb824061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 -- 172.21.15.129:0/3490240920 shutdown_connections 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.906+0000 7fb82acaa640 1 -- 172.21.15.129:0/3490240920 wait complete. 2024-03-23T15:55:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.907+0000 7fb82acaa640 1 Processor -- start 2024-03-23T15:55:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.907+0000 7fb82acaa640 1 -- start start 2024-03-23T15:55:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.908+0000 7fb82acaa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82406c0b0 0x7fb824146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.908+0000 7fb82acaa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8241544b0 0x7fb8240cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.908+0000 7fb82acaa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb82415a2e0 0x7fb8241f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.908+0000 7fb82acaa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241f0fb0 con 0x7fb82406c0b0 2024-03-23T15:55:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.908+0000 7fb82acaa640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb8241459e0 con 0x7fb8241544b0 2024-03-23T15:55:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.908+0000 7fb82acaa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb8240ce7f0 con 0x7fb82415a2e0 2024-03-23T15:55:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.909+0000 7fb823fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8241544b0 0x7fb8240cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.909+0000 7fb828a46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82406c0b0 0x7fb824146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.909+0000 7fb829247640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb82415a2e0 0x7fb8241f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.909+0000 7fb823fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8241544b0 0x7fb8240cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41666/0 (socket says 172.21.15.129:41666) 2024-03-23T15:55:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.909+0000 7fb823fff640 1 -- 172.21.15.129:0/1348776991 learned_addr learned my addr 172.21.15.129:0/1348776991 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.914+0000 7fb828a46640 1 -- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb82415a2e0 msgr2=0x7fb8241f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.914+0000 7fb828a46640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb82415a2e0 0x7fb8241f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.914+0000 7fb828a46640 1 -- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8241544b0 msgr2=0x7fb8240cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.914+0000 7fb828a46640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8241544b0 0x7fb8240cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.914+0000 7fb828a46640 1 -- 172.21.15.129:0/1348776991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb810009400 con 0x7fb82406c0b0 2024-03-23T15:55:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.917+0000 7fb828a46640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82406c0b0 0x7fb824146710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb81800c3c0 tx=0x7fb81800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.921+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8180052d0 con 0x7fb82406c0b0 2024-03-23T15:55:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.921+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb818005450 con 0x7fb82406c0b0 2024-03-23T15:55:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.921+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8240ceab0 con 0x7fb82406c0b0 2024-03-23T15:55:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.921+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818003830 con 0x7fb82406c0b0 2024-03-23T15:55:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.921+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb824145c30 con 0x7fb82406c0b0 2024-03-23T15:55:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:11.923+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb824075740 con 0x7fb82406c0b0 2024-03-23T15:55:12.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.186+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb818003990 con 0x7fb82406c0b0 2024-03-23T15:55:12.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.186+0000 7fb8217fa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb80806b890 0x7fb80806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:12.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.187+0000 7fb823fff640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb80806b890 0x7fb80806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:12.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.187+0000 7fb823fff640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb80806b890 0x7fb80806dd50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb81002ba30 tx=0x7fb810008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:12.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.209+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb818058370 con 0x7fb82406c0b0 2024-03-23T15:55:12.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.209+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb818054610 con 0x7fb82406c0b0 2024-03-23T15:55:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.427+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb824157d00 con 0x7fb82406c0b0 2024-03-23T15:55:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.442+0000 7fb8217fa640 1 -- 172.21.15.129:0/1348776991 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb818017020 con 0x7fb82406c0b0 2024-03-23T15:55:12.445 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:12.445 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:12.445 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb80806b890 msgr2=0x7fb80806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb80806b890 0x7fb80806dd50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb81002ba30 tx=0x7fb810008030 comp rx=0 tx=0).stop 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82406c0b0 msgr2=0x7fb824146710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82406c0b0 0x7fb824146710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb81800c3c0 tx=0x7fb81800c880 comp rx=0 tx=0).stop 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 shutdown_connections 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb82415a2e0 0x7fb8241f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb80806b890 0x7fb80806dd50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8241544b0 0x7fb8240cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 --2- 172.21.15.129:0/1348776991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82406c0b0 0x7fb824146710 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 >> 172.21.15.129:0/1348776991 conn(0x7fb824061b50 msgr2=0x7fb824158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 shutdown_connections 2024-03-23T15:55:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:12.444+0000 7fb82acaa640 1 -- 172.21.15.129:0/1348776991 wait complete. 2024-03-23T15:55:13.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:13.062 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:14.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:14.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.165+0000 7f1ff5c55640 1 Processor -- start 2024-03-23T15:55:14.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.165+0000 7f1ff5c55640 1 -- start start 2024-03-23T15:55:14.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.166+0000 7f1ff5c55640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:14.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.166+0000 7f1ff5c55640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff0070640 con 0x7f1ff015dba0 2024-03-23T15:55:14.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.166+0000 7f1ff5c55640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff00c4c00 con 0x7f1ff006cf70 2024-03-23T15:55:14.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.166+0000 7f1ff5c55640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff00ccec0 con 0x7f1ff00cca30 2024-03-23T15:55:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.167+0000 7f1feeffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1ff006cf70 0x7f1ff015df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33916/0 (socket says 172.21.15.129:33916) 2024-03-23T15:55:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.167+0000 7f1feeffd640 1 -- 172.21.15.129:0/2829305669 learned_addr learned my addr 172.21.15.129:0/2829305669 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:14.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.167+0000 7f1feffff640 1 --2- 172.21.15.129:0/2829305669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:14.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.170+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2156818416 0 0) 0x7f1ff0070640 con 0x7f1ff015dba0 2024-03-23T15:55:14.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.170+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fd8003520 con 0x7f1ff015dba0 2024-03-23T15:55:14.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.170+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1318930084 0 0) 0x7f1fe4001160 con 0x7f1ff006cf70 2024-03-23T15:55:14.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.170+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ff0070640 con 0x7f1ff006cf70 2024-03-23T15:55:14.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.171+0000 7f1feffff640 1 -- 172.21.15.129:0/2829305669 >> v1:172.21.15.157:6789/0 conn(0x7f1ff006cf70 legacy=0x7f1ff015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.171+0000 7f1feffff640 1 -- 172.21.15.129:0/2829305669 >> v1:172.21.15.129:6789/0 conn(0x7f1ff015dba0 legacy=0x7f1ff006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.171+0000 7f1feffff640 1 -- 172.21.15.129:0/2829305669 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff006c240 con 0x7f1ff00cca30 2024-03-23T15:55:14.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.174+0000 7f1feffff640 1 --2- 172.21.15.129:0/2829305669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01540a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009df0 tx=0x7f1fe802bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4507b98068296c67 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:14.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.179+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe802cd40 con 0x7f1ff00cca30 2024-03-23T15:55:14.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.179+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe8034440 con 0x7f1ff00cca30 2024-03-23T15:55:14.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.179+0000 7f1fee7fc640 1 -- 172.21.15.129:0/2829305669 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe802cea0 con 0x7f1ff00cca30 2024-03-23T15:55:14.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 -- 172.21.15.129:0/2829305669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 msgr2=0x7f1ff01540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/2829305669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01540a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009df0 tx=0x7f1fe802bf90 comp rx=0 tx=0).stop 2024-03-23T15:55:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 -- 172.21.15.129:0/2829305669 shutdown_connections 2024-03-23T15:55:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/2829305669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01540a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 -- 172.21.15.129:0/2829305669 >> 172.21.15.129:0/2829305669 conn(0x7f1ff0071300 msgr2=0x7f1ff00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 -- 172.21.15.129:0/2829305669 shutdown_connections 2024-03-23T15:55:14.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.180+0000 7f1ff5c55640 1 -- 172.21.15.129:0/2829305669 wait complete. 2024-03-23T15:55:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.182+0000 7f1ff5c55640 1 Processor -- start 2024-03-23T15:55:14.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.182+0000 7f1ff5c55640 1 -- start start 2024-03-23T15:55:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1ff5c55640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ff006cf70 0x7f1ff01f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1ff5c55640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1ff5c55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ff015dba0 0x7f1ff0152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1ff5c55640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01579c0 con 0x7f1ff015dba0 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1ff5c55640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01f1540 con 0x7f1ff006cf70 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1ff5c55640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff00c4fa0 con 0x7f1ff00cca30 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.183+0000 7f1fef7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ff006cf70 0x7f1ff01f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.184+0000 7f1feffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ff015dba0 0x7f1ff0152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.184+0000 7f1feeffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.184+0000 7f1fef7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ff006cf70 0x7f1ff01f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41682/0 (socket says 172.21.15.129:41682) 2024-03-23T15:55:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.184+0000 7f1fef7fe640 1 -- 172.21.15.129:0/74097899 learned_addr learned my addr 172.21.15.129:0/74097899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:14.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.187+0000 7f1feffff640 1 -- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 msgr2=0x7f1ff01522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.187+0000 7f1feffff640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.187+0000 7f1feffff640 1 -- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ff006cf70 msgr2=0x7f1ff01f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.187+0000 7f1feffff640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ff006cf70 0x7f1ff01f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.187+0000 7f1feffff640 1 -- 172.21.15.129:0/74097899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe8009a70 con 0x7f1ff015dba0 2024-03-23T15:55:14.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.190+0000 7f1feffff640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ff015dba0 0x7f1ff0152830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8033000 tx=0x7f1fe8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.193+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8040450 con 0x7f1ff015dba0 2024-03-23T15:55:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.193+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe80405b0 con 0x7f1ff015dba0 2024-03-23T15:55:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.193+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8040960 con 0x7f1ff015dba0 2024-03-23T15:55:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.193+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff015c2a0 con 0x7f1ff015dba0 2024-03-23T15:55:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.193+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff015c840 con 0x7f1ff015dba0 2024-03-23T15:55:14.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.194+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff00686f0 con 0x7f1ff015dba0 2024-03-23T15:55:14.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.452+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1fe8040ac0 con 0x7f1ff015dba0 2024-03-23T15:55:14.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.452+0000 7f1fd7fff640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fdc06b810 0x7f1fdc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:14.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.453+0000 7f1fef7fe640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fdc06b810 0x7f1fdc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:14.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.453+0000 7f1fef7fe640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fdc06b810 0x7f1fdc06dcd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00017d0 tx=0x7f1fe0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:14.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.467+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1fe807bae0 con 0x7f1ff015dba0 2024-03-23T15:55:14.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.467+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1fe8077d80 con 0x7f1ff015dba0 2024-03-23T15:55:14.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.714+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ff01f16f0 con 0x7f1ff015dba0 2024-03-23T15:55:14.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.747+0000 7f1fd7fff640 1 -- 172.21.15.129:0/74097899 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1ff01f16f0 con 0x7f1ff015dba0 2024-03-23T15:55:14.750 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:14.751 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:14.751 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:14.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fdc06b810 msgr2=0x7f1fdc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fdc06b810 0x7f1fdc06dcd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00017d0 tx=0x7f1fe0009040 comp rx=0 tx=0).stop 2024-03-23T15:55:14.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ff015dba0 msgr2=0x7f1ff0152830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:14.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ff015dba0 0x7f1ff0152830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8033000 tx=0x7f1fe8004a40 comp rx=0 tx=0).stop 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 shutdown_connections 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ff015dba0 0x7f1ff0152830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ff00cca30 0x7f1ff01522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1fdc06b810 0x7f1fdc06dcd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 --2- 172.21.15.129:0/74097899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ff006cf70 0x7f1ff01f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 >> 172.21.15.129:0/74097899 conn(0x7f1ff0071300 msgr2=0x7f1ff0153290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 shutdown_connections 2024-03-23T15:55:14.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:14.750+0000 7f1ff5c55640 1 -- 172.21.15.129:0/74097899 wait complete. 2024-03-23T15:55:15.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:15.370 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:16.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:16.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.472+0000 7f69df56c640 1 Processor -- start 2024-03-23T15:55:16.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.473+0000 7f69df56c640 1 -- start start 2024-03-23T15:55:16.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.474+0000 7f69df56c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f69d8150d00 0x7f69d81568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:16.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.474+0000 7f69df56c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8156df0 0x7f69d806d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:16.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.474+0000 7f69df56c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d8073560 con 0x7f69d8072ad0 2024-03-23T15:55:16.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.474+0000 7f69df56c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f69d81510e0 con 0x7f69d8150d00 2024-03-23T15:55:16.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.474+0000 7f69df56c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f69d8072fc0 con 0x7f69d8156df0 2024-03-23T15:55:16.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.474+0000 7f69ddb09640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f69d8072ad0 0x7f69d8159e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44136/0 (socket says 172.21.15.129:44136) 2024-03-23T15:55:16.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.475+0000 7f69ddb09640 1 -- 172.21.15.129:0/1375083898 learned_addr learned my addr 172.21.15.129:0/1375083898 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:16.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.475+0000 7f69dcb07640 1 --2- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8156df0 0x7f69d806d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:16.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.475+0000 7f69dd308640 1 --2- 172.21.15.129:0/1375083898 >> v2:172.21.15.157:3300/0 conn(0x7f69d8150d00 0x7f69d81568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:16.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.477+0000 7f69cffff640 1 -- 172.21.15.129:0/1375083898 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2551454170 0 0) 0x7f69d8073560 con 0x7f69d8072ad0 2024-03-23T15:55:16.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.478+0000 7f69cffff640 1 -- 172.21.15.129:0/1375083898 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c4005730 con 0x7f69d8072ad0 2024-03-23T15:55:16.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.478+0000 7f69dd308640 1 -- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8156df0 msgr2=0x7f69d806d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:16.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.478+0000 7f69dd308640 1 --2- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8156df0 0x7f69d806d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:16.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.478+0000 7f69dd308640 1 -- 172.21.15.129:0/1375083898 >> v1:172.21.15.129:6789/0 conn(0x7f69d8072ad0 legacy=0x7f69d8159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:16.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.478+0000 7f69dd308640 1 -- 172.21.15.129:0/1375083898 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d8073280 con 0x7f69d8150d00 2024-03-23T15:55:16.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.481+0000 7f69dd308640 1 --2- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8150d00 0x7f69d81568b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f69d40097a0 tx=0x7f69d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d2462b54b082afb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:16.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.485+0000 7f69cffff640 1 -- 172.21.15.129:0/1375083898 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4031830 con 0x7f69d8150d00 2024-03-23T15:55:16.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.485+0000 7f69cffff640 1 -- 172.21.15.129:0/1375083898 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d4031990 con 0x7f69d8150d00 2024-03-23T15:55:16.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.485+0000 7f69cffff640 1 -- 172.21.15.129:0/1375083898 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d4031d20 con 0x7f69d8150d00 2024-03-23T15:55:16.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 -- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8150d00 msgr2=0x7f69d81568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:16.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 --2- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8150d00 0x7f69d81568b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f69d40097a0 tx=0x7f69d402b8d0 comp rx=0 tx=0).stop 2024-03-23T15:55:16.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 -- 172.21.15.129:0/1375083898 shutdown_connections 2024-03-23T15:55:16.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 --2- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8156df0 0x7f69d806d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:16.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 --2- 172.21.15.129:0/1375083898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8150d00 0x7f69d81568b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:16.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 -- 172.21.15.129:0/1375083898 >> 172.21.15.129:0/1375083898 conn(0x7f69d80c8370 msgr2=0x7f69d80c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:16.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 -- 172.21.15.129:0/1375083898 shutdown_connections 2024-03-23T15:55:16.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.486+0000 7f69df56c640 1 -- 172.21.15.129:0/1375083898 wait complete. 2024-03-23T15:55:16.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.488+0000 7f69df56c640 1 Processor -- start 2024-03-23T15:55:16.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.488+0000 7f69df56c640 1 -- start start 2024-03-23T15:55:16.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.489+0000 7f69df56c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8072ad0 0x7f69d8148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:16.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.489+0000 7f69df56c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 0x7f69d8149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:16.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.489+0000 7f69df56c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8156df0 0x7f69d814a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:16.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.489+0000 7f69df56c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69d81ec4a0 con 0x7f69d8150d00 2024-03-23T15:55:16.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.489+0000 7f69df56c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f69d8154ac0 con 0x7f69d8156df0 2024-03-23T15:55:16.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.489+0000 7f69df56c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f69d81eb660 con 0x7f69d8072ad0 2024-03-23T15:55:16.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.490+0000 7f69dcb07640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 0x7f69d8149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:16.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.490+0000 7f69dd308640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8072ad0 0x7f69d8148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:16.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.490+0000 7f69ddb09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8156df0 0x7f69d814a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:16.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.490+0000 7f69dcb07640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 0x7f69d8149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50428/0 (socket says 172.21.15.129:50428) 2024-03-23T15:55:16.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.490+0000 7f69dcb07640 1 -- 172.21.15.129:0/3134064209 learned_addr learned my addr 172.21.15.129:0/3134064209 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:16.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.494+0000 7f69dcb07640 1 -- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8072ad0 msgr2=0x7f69d8148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:16.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.494+0000 7f69dcb07640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8072ad0 0x7f69d8148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:16.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.494+0000 7f69dcb07640 1 -- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8156df0 msgr2=0x7f69d814a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:16.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.494+0000 7f69dcb07640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8156df0 0x7f69d814a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:16.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.494+0000 7f69dcb07640 1 -- 172.21.15.129:0/3134064209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d4009400 con 0x7f69d8150d00 2024-03-23T15:55:16.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.497+0000 7f69dcb07640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 0x7f69d8149e50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f69c800e840 tx=0x7f69c800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:16.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.500+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c80094f0 con 0x7f69d8150d00 2024-03-23T15:55:16.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.500+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c800c940 con 0x7f69d8150d00 2024-03-23T15:55:16.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.501+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d81eb920 con 0x7f69d8150d00 2024-03-23T15:55:16.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.501+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c800ccd0 con 0x7f69d8150d00 2024-03-23T15:55:16.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.501+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d81ebee0 con 0x7f69d8150d00 2024-03-23T15:55:16.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.502+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d8075740 con 0x7f69d8150d00 2024-03-23T15:55:16.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.521+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f69c8013470 con 0x7f69d8150d00 2024-03-23T15:55:16.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.522+0000 7f69cdffb640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f69bc06b810 0x7f69bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:16.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.522+0000 7f69dd308640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f69bc06b810 0x7f69bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:16.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.522+0000 7f69dd308640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f69bc06b810 0x7f69bc06dcd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f69d4036000 tx=0x7f69d403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.768+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f69c8014e40 con 0x7f69d8150d00 2024-03-23T15:55:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:16.768+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f69c8055500 con 0x7f69d8150d00 2024-03-23T15:55:17.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.005+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69d8154890 con 0x7f69d8150d00 2024-03-23T15:55:17.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.067+0000 7f69cdffb640 1 -- 172.21.15.129:0/3134064209 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f69c8021020 con 0x7f69d8150d00 2024-03-23T15:55:17.070 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:17.070 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f69bc06b810 msgr2=0x7f69bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f69bc06b810 0x7f69bc06dcd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f69d4036000 tx=0x7f69d403c040 comp rx=0 tx=0).stop 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 msgr2=0x7f69d8149e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 0x7f69d8149e50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f69c800e840 tx=0x7f69c800ed00 comp rx=0 tx=0).stop 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 shutdown_connections 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f69bc06b810 0x7f69bc06dcd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69d8156df0 0x7f69d814a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d8150d00 0x7f69d8149e50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 --2- 172.21.15.129:0/3134064209 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69d8072ad0 0x7f69d8148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 >> 172.21.15.129:0/3134064209 conn(0x7f69d80c8370 msgr2=0x7f69d80c5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 shutdown_connections 2024-03-23T15:55:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:17.070+0000 7f69df56c640 1 -- 172.21.15.129:0/3134064209 wait complete. 2024-03-23T15:55:17.687 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:17.687 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:18.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:18.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.788+0000 7f9da89a2640 1 Processor -- start 2024-03-23T15:55:18.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 -- start start 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9da41544b0 0x7f9da40c8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9da40c9180 0x7f9da406d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9da415a440 con 0x7f9da41544b0 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9da415a720 con 0x7f9da40c9180 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.789+0000 7f9da89a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9da415aa00 con 0x7f9da40c9760 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.790+0000 7f9da37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9da41544b0 0x7f9da40c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.790+0000 7f9da3fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9da40c9180 0x7f9da406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.790+0000 7f9da2ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:18.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.790+0000 7f9da37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9da41544b0 0x7f9da40c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50430/0 (socket says 172.21.15.129:50430) 2024-03-23T15:55:18.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.790+0000 7f9da37fe640 1 -- 172.21.15.129:0/2753261684 learned_addr learned my addr 172.21.15.129:0/2753261684 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.792+0000 7f9da37fe640 1 -- 172.21.15.129:0/2753261684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 msgr2=0x7f9da40c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.792+0000 7f9da37fe640 1 --2- 172.21.15.129:0/2753261684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.793+0000 7f9da37fe640 1 -- 172.21.15.129:0/2753261684 >> v2:172.21.15.157:3300/0 conn(0x7f9da40c9180 msgr2=0x7f9da406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.793+0000 7f9da37fe640 1 --2- 172.21.15.129:0/2753261684 >> v2:172.21.15.157:3300/0 conn(0x7f9da40c9180 0x7f9da406d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.793+0000 7f9da37fe640 1 -- 172.21.15.129:0/2753261684 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da415ace0 con 0x7f9da41544b0 2024-03-23T15:55:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.794+0000 7f9da37fe640 1 --2- 172.21.15.129:0/2753261684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40c8370 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9da4073590 tx=0x7f9d9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b4b004c44e8100 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.798+0000 7f9da27fc640 1 -- 172.21.15.129:0/2753261684 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94031870 con 0x7f9da41544b0 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.798+0000 7f9da27fc640 1 -- 172.21.15.129:0/2753261684 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d940319d0 con 0x7f9da41544b0 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.798+0000 7f9da27fc640 1 -- 172.21.15.129:0/2753261684 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94031d60 con 0x7f9da41544b0 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 -- 172.21.15.129:0/2753261684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 msgr2=0x7f9da40c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 --2- 172.21.15.129:0/2753261684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40c8370 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9da4073590 tx=0x7f9d9402b910 comp rx=0 tx=0).stop 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 -- 172.21.15.129:0/2753261684 shutdown_connections 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 --2- 172.21.15.129:0/2753261684 >> v2:172.21.15.157:3300/0 conn(0x7f9da40c9180 0x7f9da406d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 --2- 172.21.15.129:0/2753261684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 --2- 172.21.15.129:0/2753261684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40c8370 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 -- 172.21.15.129:0/2753261684 >> 172.21.15.129:0/2753261684 conn(0x7f9da40711c0 msgr2=0x7f9da40715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 -- 172.21.15.129:0/2753261684 shutdown_connections 2024-03-23T15:55:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.799+0000 7f9da89a2640 1 -- 172.21.15.129:0/2753261684 wait complete. 2024-03-23T15:55:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.800+0000 7f9da89a2640 1 Processor -- start 2024-03-23T15:55:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.800+0000 7f9da89a2640 1 -- start start 2024-03-23T15:55:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.801+0000 7f9da89a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9da40c9180 0x7f9da40b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.801+0000 7f9da89a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.801+0000 7f9da89a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.801+0000 7f9da89a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9da40af520 con 0x7f9da41544b0 2024-03-23T15:55:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.801+0000 7f9da89a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9da40cb830 con 0x7f9da40c9180 2024-03-23T15:55:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.801+0000 7f9da89a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9da40ae8f0 con 0x7f9da40c9760 2024-03-23T15:55:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.802+0000 7f9da3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.802+0000 7f9da37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9da40c9180 0x7f9da40b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.802+0000 7f9da2ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.802+0000 7f9da37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9da40c9180 0x7f9da40b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41706/0 (socket says 172.21.15.129:41706) 2024-03-23T15:55:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.802+0000 7f9da37fe640 1 -- 172.21.15.129:0/531685279 learned_addr learned my addr 172.21.15.129:0/531685279 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.805+0000 7f9da3fff640 1 -- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 msgr2=0x7f9da40aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.805+0000 7f9da3fff640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.805+0000 7f9da3fff640 1 -- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9da40c9180 msgr2=0x7f9da40b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.805+0000 7f9da3fff640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9da40c9180 0x7f9da40b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.805+0000 7f9da3fff640 1 -- 172.21.15.129:0/531685279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d940094d0 con 0x7f9da41544b0 2024-03-23T15:55:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.808+0000 7f9da3fff640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40ce0d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00eed0 tx=0x7f9d9c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.810+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c010040 con 0x7f9da41544b0 2024-03-23T15:55:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.810+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da41f1d60 con 0x7f9da41544b0 2024-03-23T15:55:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.810+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da40ae330 con 0x7f9da41544b0 2024-03-23T15:55:18.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.811+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da41f1ef0 con 0x7f9da41544b0 2024-03-23T15:55:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.813+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d9c009e60 con 0x7f9da41544b0 2024-03-23T15:55:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:18.813+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c01d600 con 0x7f9da41544b0 2024-03-23T15:55:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.065+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9d9c01d760 con 0x7f9da41544b0 2024-03-23T15:55:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.065+0000 7f9d8bfff640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d9006b840 0x7f9d9006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.066+0000 7f9da37fe640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d9006b840 0x7f9d9006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.066+0000 7f9da37fe640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d9006b840 0x7f9d9006dd00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d94034040 tx=0x7f9d9403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:19.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.082+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9d9c059840 con 0x7f9da41544b0 2024-03-23T15:55:19.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.082+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9d9c055ae0 con 0x7f9da41544b0 2024-03-23T15:55:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.325+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9da40cac20 con 0x7f9da41544b0 2024-03-23T15:55:19.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.337+0000 7f9d8bfff640 1 -- 172.21.15.129:0/531685279 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9d9c0591a0 con 0x7f9da41544b0 2024-03-23T15:55:19.340 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:19.341 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:19.341 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d9006b840 msgr2=0x7f9d9006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d9006b840 0x7f9d9006dd00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d94034040 tx=0x7f9d9403c040 comp rx=0 tx=0).stop 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 msgr2=0x7f9da40ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40ce0d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00eed0 tx=0x7f9d9c00b480 comp rx=0 tx=0).stop 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 shutdown_connections 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da41544b0 0x7f9da40ce0d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9da40c9760 0x7f9da40aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d9006b840 0x7f9d9006dd00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 --2- 172.21.15.129:0/531685279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9da40c9180 0x7f9da40b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:19.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 >> 172.21.15.129:0/531685279 conn(0x7f9da40711c0 msgr2=0x7f9da40c34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:19.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 shutdown_connections 2024-03-23T15:55:19.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:19.340+0000 7f9da89a2640 1 -- 172.21.15.129:0/531685279 wait complete. 2024-03-23T15:55:19.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:19.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:20.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:21.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.059+0000 7fc62fe8f640 1 Processor -- start 2024-03-23T15:55:21.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.059+0000 7fc62fe8f640 1 -- start start 2024-03-23T15:55:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.060+0000 7fc62fe8f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc6280c6000 0x7fc6280c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.060+0000 7fc62fe8f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6280cbcb0 con 0x7fc62815a2e0 2024-03-23T15:55:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.060+0000 7fc62fe8f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62815b280 con 0x7fc6281544b0 2024-03-23T15:55:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.060+0000 7fc62fe8f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc6280c5be0 con 0x7fc6280c6000 2024-03-23T15:55:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.061+0000 7fc62dc2b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc6281544b0 0x7fc6280caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60384/0 (socket says 172.21.15.129:60384) 2024-03-23T15:55:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.061+0000 7fc62dc2b640 1 -- 172.21.15.129:0/3368397618 learned_addr learned my addr 172.21.15.129:0/3368397618 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:21.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.063+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973852095 0 0) 0x7fc62815b280 con 0x7fc6281544b0 2024-03-23T15:55:21.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.063+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc614003520 con 0x7fc6281544b0 2024-03-23T15:55:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1934440921 0 0) 0x7fc614003520 con 0x7fc6281544b0 2024-03-23T15:55:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 >> v2:172.21.15.129:3300/0 conn(0x7fc6280c6000 msgr2=0x7fc6280c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 --2- 172.21.15.129:0/3368397618 >> v2:172.21.15.129:3300/0 conn(0x7fc6280c6000 0x7fc6280c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 >> v1:172.21.15.129:6789/0 conn(0x7fc62815a2e0 legacy=0x7fc62806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62815a9e0 con 0x7fc6281544b0 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7fc624001670 con 0x7fc6281544b0 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 >> v1:172.21.15.157:6789/0 conn(0x7fc6281544b0 legacy=0x7fc6280caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6140039c0 0x7fc614003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc614004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6140052a0 0x7fc61400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280c5be0 con 0x7fc6140039c0 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc61400c1e0 con 0x7fc6140052a0 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.067+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc61400c3a0 con 0x7fc6140047c0 2024-03-23T15:55:21.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.068+0000 7fc62d42a640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc614004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:21.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.071+0000 7fc62d42a640 1 -- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6140052a0 msgr2=0x7fc61400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:21.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.071+0000 7fc62d42a640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6140052a0 0x7fc61400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.071+0000 7fc62d42a640 1 -- 172.21.15.129:0/3368397618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6140039c0 msgr2=0x7fc614003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:21.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.071+0000 7fc62d42a640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6140039c0 0x7fc614003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.071+0000 7fc62d42a640 1 -- 172.21.15.129:0/3368397618 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc614003520 con 0x7fc6140047c0 2024-03-23T15:55:21.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.071+0000 7fc62e42c640 1 -- 172.21.15.129:0/3368397618 reap_dead start 2024-03-23T15:55:21.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.074+0000 7fc62d42a640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc614004c40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc618001620 tx=0x7fc6140057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2673bf4dd85a9fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:21.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.077+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc618005070 con 0x7fc6140047c0 2024-03-23T15:55:21.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.079+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61802e070 con 0x7fc6140047c0 2024-03-23T15:55:21.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.079+0000 7fc62cc29640 1 -- 172.21.15.129:0/3368397618 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc618005430 con 0x7fc6140047c0 2024-03-23T15:55:21.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 -- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 msgr2=0x7fc614004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc614004c40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc618001620 tx=0x7fc6140057f0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 -- 172.21.15.129:0/3368397618 shutdown_connections 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/3368397618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc614004c40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 -- 172.21.15.129:0/3368397618 >> 172.21.15.129:0/3368397618 conn(0x7fc628061b50 msgr2=0x7fc628061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 -- 172.21.15.129:0/3368397618 shutdown_connections 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.080+0000 7fc62fe8f640 1 -- 172.21.15.129:0/3368397618 wait complete. 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.081+0000 7fc62fe8f640 1 Processor -- start 2024-03-23T15:55:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.081+0000 7fc62fe8f640 1 -- start start 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62fe8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc628146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62fe8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc628151ab0 0x7fc6281556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62fe8f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6280cf150 0x7fc6280cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62fe8f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc628147800 con 0x7fc628151ab0 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62fe8f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc6280cecc0 con 0x7fc6280cf150 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62fe8f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc6280ce7d0 con 0x7fc6140047c0 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62d42a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc628151ab0 0x7fc6281556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62e42c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6280cf150 0x7fc6280cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62dc2b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc628146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:21.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62e42c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6280cf150 0x7fc6280cf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44260/0 (socket says 172.21.15.129:44260) 2024-03-23T15:55:21.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.082+0000 7fc62e42c640 1 -- 172.21.15.129:0/826845364 learned_addr learned my addr 172.21.15.129:0/826845364 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:21.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.086+0000 7fc62d42a640 1 -- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 msgr2=0x7fc628146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.086+0000 7fc62d42a640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc628146dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.086+0000 7fc62d42a640 1 -- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6280cf150 msgr2=0x7fc6280cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.086+0000 7fc62d42a640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6280cf150 0x7fc6280cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.086+0000 7fc62d42a640 1 -- 172.21.15.129:0/826845364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6280ce510 con 0x7fc628151ab0 2024-03-23T15:55:21.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.089+0000 7fc62d42a640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc628151ab0 0x7fc6281556f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc618030000 tx=0x7fc618000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.092+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc618007240 con 0x7fc628151ab0 2024-03-23T15:55:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.092+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc628156290 con 0x7fc628151ab0 2024-03-23T15:55:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.092+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc628156850 con 0x7fc628151ab0 2024-03-23T15:55:21.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.093+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc628075740 con 0x7fc628151ab0 2024-03-23T15:55:21.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.094+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6180073a0 con 0x7fc628151ab0 2024-03-23T15:55:21.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.094+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6180067b0 con 0x7fc628151ab0 2024-03-23T15:55:21.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.386+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc618030030 con 0x7fc628151ab0 2024-03-23T15:55:21.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.386+0000 7fc61e7fc640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc60c06b810 0x7fc60c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:21.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.386+0000 7fc62dc2b640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc60c06b810 0x7fc60c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:21.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.387+0000 7fc62dc2b640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc60c06b810 0x7fc60c06dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc624000fb0 tx=0x7fc624001a60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.405+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc618079300 con 0x7fc628151ab0 2024-03-23T15:55:21.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.405+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6180755a0 con 0x7fc628151ab0 2024-03-23T15:55:21.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.647+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc628157d00 con 0x7fc628151ab0 2024-03-23T15:55:21.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.662+0000 7fc61e7fc640 1 -- 172.21.15.129:0/826845364 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc618078c60 con 0x7fc628151ab0 2024-03-23T15:55:21.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:21.665 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:21.665 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:21.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc60c06b810 msgr2=0x7fc60c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:21.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc60c06b810 0x7fc60c06dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc624000fb0 tx=0x7fc624001a60 comp rx=0 tx=0).stop 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc628151ab0 msgr2=0x7fc6281556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc628151ab0 0x7fc6281556f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc618030000 tx=0x7fc618000fb0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 shutdown_connections 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc60c06b810 0x7fc60c06dcd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6280cf150 0x7fc6280cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc628151ab0 0x7fc6281556f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 --2- 172.21.15.129:0/826845364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6140047c0 0x7fc628146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.664+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 >> 172.21.15.129:0/826845364 conn(0x7fc628061b50 msgr2=0x7fc628159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.665+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 shutdown_connections 2024-03-23T15:55:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:21.665+0000 7fc62fe8f640 1 -- 172.21.15.129:0/826845364 wait complete. 2024-03-23T15:55:22.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:22.283 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:23.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 Processor -- start 2024-03-23T15:55:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 -- start start 2024-03-23T15:55:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb0ec15a2e0 0x7fb0ec06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec06c0b0 0x7fb0ec0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0ec0cbcb0 con 0x7fb0ec1544b0 2024-03-23T15:55:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb0ec0c5be0 con 0x7fb0ec15a2e0 2024-03-23T15:55:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.385+0000 7fb0f156c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb0ec15b280 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.387+0000 7fb0eaffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb0ec1544b0 0x7fb0ec0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60396/0 (socket says 172.21.15.129:60396) 2024-03-23T15:55:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.387+0000 7fb0eaffd640 1 -- 172.21.15.129:0/3647864726 learned_addr learned my addr 172.21.15.129:0/3647864726 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.388+0000 7fb0ea7fc640 1 --2- 172.21.15.129:0/3647864726 >> v2:172.21.15.157:3300/0 conn(0x7fb0ec15a2e0 0x7fb0ec06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:23.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.388+0000 7fb0eb7fe640 1 --2- 172.21.15.129:0/3647864726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec06c0b0 0x7fb0ec0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:23.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.392+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/3647864726 >> v2:172.21.15.157:3300/0 conn(0x7fb0ec15a2e0 msgr2=0x7fb0ec06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:23.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.392+0000 7fb0eb7fe640 1 --2- 172.21.15.129:0/3647864726 >> v2:172.21.15.157:3300/0 conn(0x7fb0ec15a2e0 0x7fb0ec06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.392+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/3647864726 >> v1:172.21.15.157:6789/0 conn(0x7fb0ec1544b0 legacy=0x7fb0ec0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:23.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.392+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/3647864726 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ec15a9e0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.394+0000 7fb0eb7fe640 1 --2- 172.21.15.129:0/3647864726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec06c0b0 0x7fb0ec0c6000 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009d90 tx=0x7fb0e402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ead61a57718483c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:23.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.396+0000 7fb0e9ffb640 1 -- 172.21.15.129:0/3647864726 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e402cce0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0e9ffb640 1 -- 172.21.15.129:0/3647864726 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0e402ce40 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0e9ffb640 1 -- 172.21.15.129:0/3647864726 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4034660 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 -- 172.21.15.129:0/3647864726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec06c0b0 msgr2=0x7fb0ec0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:23.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 --2- 172.21.15.129:0/3647864726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec06c0b0 0x7fb0ec0c6000 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009d90 tx=0x7fb0e402bf30 comp rx=0 tx=0).stop 2024-03-23T15:55:23.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 -- 172.21.15.129:0/3647864726 shutdown_connections 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 --2- 172.21.15.129:0/3647864726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec06c0b0 0x7fb0ec0c6000 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 --2- 172.21.15.129:0/3647864726 >> v2:172.21.15.157:3300/0 conn(0x7fb0ec15a2e0 0x7fb0ec06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 -- 172.21.15.129:0/3647864726 >> 172.21.15.129:0/3647864726 conn(0x7fb0ec061b50 msgr2=0x7fb0ec061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 -- 172.21.15.129:0/3647864726 shutdown_connections 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.399+0000 7fb0f156c640 1 -- 172.21.15.129:0/3647864726 wait complete. 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 Processor -- start 2024-03-23T15:55:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 -- start start 2024-03-23T15:55:23.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 0x7fb0ec0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:23.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec1544b0 0x7fb0ec0d0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0ec15a2e0 0x7fb0ec0cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec1f0e80 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec1f0020 con 0x7fb0ec15a2e0 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.401+0000 7fb0f156c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb0ec1f0300 con 0x7fb0ec1544b0 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.402+0000 7fb0ea7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec1544b0 0x7fb0ec0d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.402+0000 7fb0eb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0ec15a2e0 0x7fb0ec0cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.402+0000 7fb0eaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 0x7fb0ec0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.402+0000 7fb0eaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 0x7fb0ec0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58762/0 (socket says 172.21.15.129:58762) 2024-03-23T15:55:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.402+0000 7fb0eaffd640 1 -- 172.21.15.129:0/2794881381 learned_addr learned my addr 172.21.15.129:0/2794881381 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:23.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.406+0000 7fb0eaffd640 1 -- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec1544b0 msgr2=0x7fb0ec0d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:23.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.406+0000 7fb0eaffd640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec1544b0 0x7fb0ec0d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.406+0000 7fb0eaffd640 1 -- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0ec15a2e0 msgr2=0x7fb0ec0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:23.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.406+0000 7fb0eaffd640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0ec15a2e0 0x7fb0ec0cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.406+0000 7fb0eaffd640 1 -- 172.21.15.129:0/2794881381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e4009a70 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.408+0000 7fb0eaffd640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 0x7fb0ec0b4970 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00c400 tx=0x7fb0dc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:23.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.411+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc005ce0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.412+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0dc005e40 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.412+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc0072d0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.412+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0ec1f05e0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.412+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0ec1f0ba0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.413+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0ec075740 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.441+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb0dc007430 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.441+0000 7fb0d37fe640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0d806b890 0x7fb0d806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:23.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.442+0000 7fb0ea7fc640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0d806b890 0x7fb0d806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:23.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.442+0000 7fb0ea7fc640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0d806b890 0x7fb0d806dd50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0d00d0 tx=0x7fb0e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:23.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.447+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb0dc058430 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.699+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb0dc0546d0 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.947+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0ec0ceb20 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.961+0000 7fb0d37fe640 1 -- 172.21.15.129:0/2794881381 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb0dc017020 con 0x7fb0ec06c0b0 2024-03-23T15:55:23.964 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:23.964 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:23.964 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:23.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.963+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0d806b890 msgr2=0x7fb0d806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.963+0000 7fb0f156c640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0d806b890 0x7fb0d806dd50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0d00d0 tx=0x7fb0e0008040 comp rx=0 tx=0).stop 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.963+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 msgr2=0x7fb0ec0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.963+0000 7fb0f156c640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 0x7fb0ec0b4970 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00c400 tx=0x7fb0dc00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 shutdown_connections 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0d806b890 0x7fb0d806dd50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0ec15a2e0 0x7fb0ec0cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0ec1544b0 0x7fb0ec0d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 --2- 172.21.15.129:0/2794881381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ec06c0b0 0x7fb0ec0b4970 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 >> 172.21.15.129:0/2794881381 conn(0x7fb0ec061b50 msgr2=0x7fb0ec0c63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 shutdown_connections 2024-03-23T15:55:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:23.964+0000 7fb0f156c640 1 -- 172.21.15.129:0/2794881381 wait complete. 2024-03-23T15:55:24.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:24.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:25.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.682+0000 7f5bd6d02640 1 Processor -- start 2024-03-23T15:55:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.682+0000 7f5bd6d02640 1 -- start start 2024-03-23T15:55:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.683+0000 7f5bd6d02640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.683+0000 7f5bd6d02640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5bd00cef40 0x7f5bd015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.683+0000 7f5bd6d02640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd015b290 con 0x7f5bd00ceb00 2024-03-23T15:55:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.683+0000 7f5bd6d02640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f5bd00ce800 con 0x7f5bd00cef40 2024-03-23T15:55:25.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.683+0000 7f5bd6d02640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd0154a70 con 0x7f5bd00ccd60 2024-03-23T15:55:25.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.684+0000 7f5bcffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5bd00ceb00 0x7f5bd006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60412/0 (socket says 172.21.15.129:60412) 2024-03-23T15:55:25.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.684+0000 7f5bcffff640 1 -- 172.21.15.129:0/1912555081 learned_addr learned my addr 172.21.15.129:0/1912555081 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:25.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.684+0000 7f5bd4a9e640 1 --2- 172.21.15.129:0/1912555081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.686+0000 7f5bd529f640 1 --2- 172.21.15.129:0/1912555081 >> v2:172.21.15.129:3300/0 conn(0x7f5bd00cef40 0x7f5bd015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.687+0000 7f5bcf7fe640 1 -- 172.21.15.129:0/1912555081 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482260042 0 0) 0x7f5bbc001110 con 0x7f5bd00ceb00 2024-03-23T15:55:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.687+0000 7f5bcf7fe640 1 -- 172.21.15.129:0/1912555081 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bc0003520 con 0x7f5bd00ceb00 2024-03-23T15:55:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.689+0000 7f5bd529f640 1 -- 172.21.15.129:0/1912555081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 msgr2=0x7f5bd015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.689+0000 7f5bd529f640 1 --2- 172.21.15.129:0/1912555081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.689+0000 7f5bd529f640 1 -- 172.21.15.129:0/1912555081 >> v1:172.21.15.157:6789/0 conn(0x7f5bd00ceb00 legacy=0x7f5bd006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.689+0000 7f5bd529f640 1 -- 172.21.15.129:0/1912555081 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd00cd950 con 0x7f5bd00cef40 2024-03-23T15:55:25.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.692+0000 7f5bd529f640 1 --2- 172.21.15.129:0/1912555081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00cef40 0x7f5bd015b8f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009d90 tx=0x7f5bc802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c52d9db956eec4d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:25.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.697+0000 7f5bcf7fe640 1 -- 172.21.15.129:0/1912555081 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc802cce0 con 0x7f5bd00cef40 2024-03-23T15:55:25.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.698+0000 7f5bcf7fe640 1 -- 172.21.15.129:0/1912555081 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc802ce40 con 0x7f5bd00cef40 2024-03-23T15:55:25.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.698+0000 7f5bcf7fe640 1 -- 172.21.15.129:0/1912555081 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8034660 con 0x7f5bd00cef40 2024-03-23T15:55:25.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.698+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1912555081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00cef40 msgr2=0x7f5bd015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:25.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.698+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1912555081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00cef40 0x7f5bd015b8f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009d90 tx=0x7f5bc802bf30 comp rx=0 tx=0).stop 2024-03-23T15:55:25.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.699+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1912555081 shutdown_connections 2024-03-23T15:55:25.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.699+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1912555081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00cef40 0x7f5bd015b8f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:25.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.699+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1912555081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:25.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.699+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1912555081 >> 172.21.15.129:0/1912555081 conn(0x7f5bd00b1840 msgr2=0x7f5bd00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:25.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.699+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1912555081 shutdown_connections 2024-03-23T15:55:25.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.699+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1912555081 wait complete. 2024-03-23T15:55:25.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.701+0000 7f5bd6d02640 1 Processor -- start 2024-03-23T15:55:25.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.701+0000 7f5bd6d02640 1 -- start start 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bd6d02640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd0151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bd6d02640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 0x7f5bd0159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bd6d02640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bd00cef40 0x7f5bd01595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bd6d02640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01e3880 con 0x7f5bd00ceb00 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bd6d02640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01e5570 con 0x7f5bd00cef40 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bd6d02640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd01e3db0 con 0x7f5bd00ccd60 2024-03-23T15:55:25.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.702+0000 7f5bcffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 0x7f5bd0159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:25.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.703+0000 7f5bd529f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bd00cef40 0x7f5bd01595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:25.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.703+0000 7f5bd4a9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd0151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:25.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.703+0000 7f5bcffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 0x7f5bd0159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58766/0 (socket says 172.21.15.129:58766) 2024-03-23T15:55:25.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.703+0000 7f5bcffff640 1 -- 172.21.15.129:0/1323447711 learned_addr learned my addr 172.21.15.129:0/1323447711 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.706+0000 7f5bcffff640 1 -- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 msgr2=0x7f5bd0151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.706+0000 7f5bcffff640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd0151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.706+0000 7f5bcffff640 1 -- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bd00cef40 msgr2=0x7f5bd01595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.706+0000 7f5bcffff640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bd00cef40 0x7f5bd01595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.706+0000 7f5bcffff640 1 -- 172.21.15.129:0/1323447711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc8009a70 con 0x7f5bd00ceb00 2024-03-23T15:55:25.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.710+0000 7f5bcffff640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 0x7f5bd0159070 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00c700 tx=0x7f5bbc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:25.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.714+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc005ab0 con 0x7f5bd00ceb00 2024-03-23T15:55:25.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.714+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc005c10 con 0x7f5bd00ceb00 2024-03-23T15:55:25.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.714+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd015af40 con 0x7f5bd00ceb00 2024-03-23T15:55:25.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.714+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc001d50 con 0x7f5bd00ceb00 2024-03-23T15:55:25.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.714+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd01e5200 con 0x7f5bd00ceb00 2024-03-23T15:55:25.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.715+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd004f530 con 0x7f5bd00ceb00 2024-03-23T15:55:25.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.734+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5bbc007070 con 0x7f5bd00ceb00 2024-03-23T15:55:25.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.735+0000 7f5bcd7fa640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bb406b890 0x7f5bb406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:25.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.735+0000 7f5bd4a9e640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bb406b890 0x7f5bb406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:25.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.735+0000 7f5bd4a9e640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bb406b890 0x7f5bb406dd50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009b40 tx=0x7f5bc4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:25.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.738+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5bbc0581e0 con 0x7f5bd00ceb00 2024-03-23T15:55:25.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:25.981+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5bbc054480 con 0x7f5bd00ceb00 2024-03-23T15:55:26.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.201+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bd006db50 con 0x7f5bd00ceb00 2024-03-23T15:55:26.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.221+0000 7f5bcd7fa640 1 -- 172.21.15.129:0/1323447711 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5bbc057b40 con 0x7f5bd00ceb00 2024-03-23T15:55:26.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:26.224 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:26.224 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:26.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bb406b890 msgr2=0x7f5bb406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:26.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bb406b890 0x7f5bb406dd50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009b40 tx=0x7f5bc4009190 comp rx=0 tx=0).stop 2024-03-23T15:55:26.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 msgr2=0x7f5bd0159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:26.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 0x7f5bd0159070 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00c700 tx=0x7f5bbc00cbc0 comp rx=0 tx=0).stop 2024-03-23T15:55:26.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 shutdown_connections 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5bb406b890 0x7f5bb406dd50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bd00cef40 0x7f5bd01595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bd00ceb00 0x7f5bd0159070 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 --2- 172.21.15.129:0/1323447711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bd00ccd60 0x7f5bd0151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 >> 172.21.15.129:0/1323447711 conn(0x7f5bd00b1840 msgr2=0x7f5bd0153f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 shutdown_connections 2024-03-23T15:55:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:26.223+0000 7f5bd6d02640 1 -- 172.21.15.129:0/1323447711 wait complete. 2024-03-23T15:55:26.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:26.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:27.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:27.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 Processor -- start 2024-03-23T15:55:27.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 -- start start 2024-03-23T15:55:27.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:27.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc2a006c0b0 0x7fc2a00c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:27.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a00cbcb0 con 0x7fc2a01544b0 2024-03-23T15:55:27.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a00c5be0 con 0x7fc2a006c0b0 2024-03-23T15:55:27.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.941+0000 7fc2a5855640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a015b280 con 0x7fc2a015a2e0 2024-03-23T15:55:27.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.942+0000 7fc29f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc2a006c0b0 0x7fc2a00c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:27.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.942+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:27.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.943+0000 7fc29effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc2a01544b0 0x7fc2a00caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60428/0 (socket says 172.21.15.129:60428) 2024-03-23T15:55:27.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.943+0000 7fc29effd640 1 -- 172.21.15.129:0/694501249 learned_addr learned my addr 172.21.15.129:0/694501249 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:27.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.946+0000 7fc29f7fe640 1 -- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 msgr2=0x7fc2a006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:27.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.946+0000 7fc29f7fe640 1 --2- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:27.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.946+0000 7fc29f7fe640 1 -- 172.21.15.129:0/694501249 >> v1:172.21.15.157:6789/0 conn(0x7fc2a01544b0 legacy=0x7fc2a00caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:27.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.946+0000 7fc29f7fe640 1 -- 172.21.15.129:0/694501249 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a015a9e0 con 0x7fc2a006c0b0 2024-03-23T15:55:27.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.949+0000 7fc29f7fe640 1 --2- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00c6000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc298009d90 tx=0x7fc29802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e65724f5ad1e0ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:27.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc29dffb640 1 -- 172.21.15.129:0/694501249 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29802cce0 con 0x7fc2a006c0b0 2024-03-23T15:55:27.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc29dffb640 1 -- 172.21.15.129:0/694501249 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29802ce40 con 0x7fc2a006c0b0 2024-03-23T15:55:27.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc29dffb640 1 -- 172.21.15.129:0/694501249 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298034660 con 0x7fc2a006c0b0 2024-03-23T15:55:27.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 -- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a006c0b0 msgr2=0x7fc2a00c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 --2- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00c6000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc298009d90 tx=0x7fc29802bf30 comp rx=0 tx=0).stop 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 -- 172.21.15.129:0/694501249 shutdown_connections 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 --2- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00c6000 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 --2- 172.21.15.129:0/694501249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 -- 172.21.15.129:0/694501249 >> 172.21.15.129:0/694501249 conn(0x7fc2a0061b50 msgr2=0x7fc2a0061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 -- 172.21.15.129:0/694501249 shutdown_connections 2024-03-23T15:55:27.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.955+0000 7fc2a5855640 1 -- 172.21.15.129:0/694501249 wait complete. 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 Processor -- start 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 -- start start 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 0x7fc2a00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a00ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a0143830 con 0x7fc2a006c0b0 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a00ce430 con 0x7fc2a01544b0 2024-03-23T15:55:27.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.957+0000 7fc2a5855640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a00d0830 con 0x7fc2a015a2e0 2024-03-23T15:55:27.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.958+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 0x7fc2a00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:27.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.958+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a00ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:27.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.958+0000 7fc29e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 0x7fc2a00b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44298/0 (socket says 172.21.15.129:44298) 2024-03-23T15:55:27.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.958+0000 7fc29e7fc640 1 -- 172.21.15.129:0/1559633082 learned_addr learned my addr 172.21.15.129:0/1559633082 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:27.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.958+0000 7fc29effd640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:27.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.962+0000 7fc29e7fc640 1 -- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 msgr2=0x7fc2a00ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:27.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.962+0000 7fc29e7fc640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a00ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:27.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.962+0000 7fc29e7fc640 1 -- 172.21.15.129:0/1559633082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a006c0b0 msgr2=0x7fc2a00cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:27.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.962+0000 7fc29e7fc640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:27.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.962+0000 7fc29e7fc640 1 -- 172.21.15.129:0/1559633082 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc298009a70 con 0x7fc2a01544b0 2024-03-23T15:55:27.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.964+0000 7fc29e7fc640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 0x7fc2a00b4af0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc29400e9b0 tx=0x7fc29400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:27.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.969+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc294009670 con 0x7fc2a01544b0 2024-03-23T15:55:27.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.969+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29400c9a0 con 0x7fc2a01544b0 2024-03-23T15:55:27.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.969+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29400cd50 con 0x7fc2a01544b0 2024-03-23T15:55:27.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.969+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a01423e0 con 0x7fc2a01544b0 2024-03-23T15:55:27.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.969+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a00ce0c0 con 0x7fc2a01544b0 2024-03-23T15:55:27.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:27.970+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a00ce5c0 con 0x7fc2a01544b0 2024-03-23T15:55:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.229+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc294013470 con 0x7fc2a01544b0 2024-03-23T15:55:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.229+0000 7fc287fff640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc28c06b860 0x7fc28c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.229+0000 7fc29effd640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc28c06b860 0x7fc28c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.230+0000 7fc29effd640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc28c06b860 0x7fc28c06dd20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc290007fd0 tx=0x7fc290007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.233+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc294014e40 con 0x7fc2a01544b0 2024-03-23T15:55:28.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.243+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc294055690 con 0x7fc2a01544b0 2024-03-23T15:55:28.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.462+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2a0075740 con 0x7fc2a01544b0 2024-03-23T15:55:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.478+0000 7fc287fff640 1 -- 172.21.15.129:0/1559633082 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc294017090 con 0x7fc2a01544b0 2024-03-23T15:55:28.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:28.481 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:28.481 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:28.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc28c06b860 msgr2=0x7fc28c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc28c06b860 0x7fc28c06dd20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc290007fd0 tx=0x7fc290007e80 comp rx=0 tx=0).stop 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 msgr2=0x7fc2a00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 0x7fc2a00b4af0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc29400e9b0 tx=0x7fc29400ee70 comp rx=0 tx=0).stop 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 shutdown_connections 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc2a015a2e0 0x7fc2a00ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc28c06b860 0x7fc28c06dd20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2a01544b0 0x7fc2a00b4af0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 --2- 172.21.15.129:0/1559633082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2a006c0b0 0x7fc2a00cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.480+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 >> 172.21.15.129:0/1559633082 conn(0x7fc2a0061b50 msgr2=0x7fc2a00c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.481+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 shutdown_connections 2024-03-23T15:55:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:28.481+0000 7fc2a5855640 1 -- 172.21.15.129:0/1559633082 wait complete. 2024-03-23T15:55:29.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:29.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:30.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:30.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.200+0000 7fd6d1b9a640 1 Processor -- start 2024-03-23T15:55:30.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.200+0000 7fd6d1b9a640 1 -- start start 2024-03-23T15:55:30.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.200+0000 7fd6d1b9a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fd6cc0ccd60 0x7fd6cc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:30.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.200+0000 7fd6d1b9a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd6cc0ceb00 0x7fd6cc06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:30.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.201+0000 7fd6d1b9a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6cc15b290 con 0x7fd6cc0cef40 2024-03-23T15:55:30.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.201+0000 7fd6d1b9a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fd6cc0ce800 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.201+0000 7fd6d1b9a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd6cc154a70 con 0x7fd6cc0ceb00 2024-03-23T15:55:30.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.201+0000 7fd6caffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd6cc0ceb00 0x7fd6cc06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:30.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.201+0000 7fd6cbfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd6cc0cef40 0x7fd6cc15b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:54350/0 (socket says 172.21.15.129:54350) 2024-03-23T15:55:30.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.202+0000 7fd6cbfff640 1 -- 172.21.15.129:0/1377586483 learned_addr learned my addr 172.21.15.129:0/1377586483 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:30.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.202+0000 7fd6cb7fe640 1 --2- 172.21.15.129:0/1377586483 >> v2:172.21.15.129:3300/0 conn(0x7fd6cc0ccd60 0x7fd6cc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:30.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.205+0000 7fd6caffd640 1 -- 172.21.15.129:0/1377586483 >> v2:172.21.15.129:3300/0 conn(0x7fd6cc0ccd60 msgr2=0x7fd6cc15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:30.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.205+0000 7fd6caffd640 1 --2- 172.21.15.129:0/1377586483 >> v2:172.21.15.129:3300/0 conn(0x7fd6cc0ccd60 0x7fd6cc15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.205+0000 7fd6caffd640 1 -- 172.21.15.129:0/1377586483 >> v1:172.21.15.129:6789/0 conn(0x7fd6cc0cef40 legacy=0x7fd6cc15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:30.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.205+0000 7fd6caffd640 1 -- 172.21.15.129:0/1377586483 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc0cd950 con 0x7fd6cc0ceb00 2024-03-23T15:55:30.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.208+0000 7fd6caffd640 1 --2- 172.21.15.129:0/1377586483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc06c270 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc08d1a0 tx=0x7fd6c002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a954822cb10d213e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:30.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.213+0000 7fd6ca7fc640 1 -- 172.21.15.129:0/1377586483 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0030610 con 0x7fd6cc0ceb00 2024-03-23T15:55:30.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.213+0000 7fd6ca7fc640 1 -- 172.21.15.129:0/1377586483 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c0030770 con 0x7fd6cc0ceb00 2024-03-23T15:55:30.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.213+0000 7fd6ca7fc640 1 -- 172.21.15.129:0/1377586483 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0030b00 con 0x7fd6cc0ceb00 2024-03-23T15:55:30.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.213+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/1377586483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 msgr2=0x7fd6cc06c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:30.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.213+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/1377586483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc06c270 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc08d1a0 tx=0x7fd6c002b960 comp rx=0 tx=0).stop 2024-03-23T15:55:30.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.213+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/1377586483 shutdown_connections 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.214+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/1377586483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc06c270 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.214+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/1377586483 >> v2:172.21.15.129:3300/0 conn(0x7fd6cc0ccd60 0x7fd6cc15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.214+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/1377586483 >> 172.21.15.129:0/1377586483 conn(0x7fd6cc0b1840 msgr2=0x7fd6cc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.214+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/1377586483 shutdown_connections 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.214+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/1377586483 wait complete. 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.215+0000 7fd6d1b9a640 1 Processor -- start 2024-03-23T15:55:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.215+0000 7fd6d1b9a640 1 -- start start 2024-03-23T15:55:30.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6d1b9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 0x7fd6cc1f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:30.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6d1b9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc1590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:30.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6d1b9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6cc0cef40 0x7fd6cc1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:30.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6d1b9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc0c99f0 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6d1b9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1565a0 con 0x7fd6cc0ceb00 2024-03-23T15:55:30.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6d1b9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6cc1f1230 con 0x7fd6cc0cef40 2024-03-23T15:55:30.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 0x7fd6cc1f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:30.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6caffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc1590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:30.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.216+0000 7fd6cbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6cc0cef40 0x7fd6cc1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:30.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.217+0000 7fd6cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 0x7fd6cc1f0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58802/0 (socket says 172.21.15.129:58802) 2024-03-23T15:55:30.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.217+0000 7fd6cb7fe640 1 -- 172.21.15.129:0/3627807432 learned_addr learned my addr 172.21.15.129:0/3627807432 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:30.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.221+0000 7fd6cb7fe640 1 -- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6cc0cef40 msgr2=0x7fd6cc1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:30.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.221+0000 7fd6cb7fe640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6cc0cef40 0x7fd6cc1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.221+0000 7fd6cb7fe640 1 -- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 msgr2=0x7fd6cc1590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:30.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.221+0000 7fd6cb7fe640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc1590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.221+0000 7fd6cb7fe640 1 -- 172.21.15.129:0/3627807432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c0009400 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.224+0000 7fd6cb7fe640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 0x7fd6cc1f0cf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00e870 tx=0x7fd6bc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:30.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.228+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0094f0 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.228+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6bc00c940 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.228+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc155c90 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.228+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00ccf0 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.228+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1e42c0 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.229+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc156810 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.513+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd6bc013470 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.513+0000 7fd6b3fff640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd6b806b810 0x7fd6b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:30.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.514+0000 7fd6caffd640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd6b806b810 0x7fd6b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:30.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.514+0000 7fd6caffd640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd6b806b810 0x7fd6b806dcd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0036000 tx=0x7fd6c0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:30.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.532+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd6bc014e40 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.532+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd6bc0554b0 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.784+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6cc06db50 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.797+0000 7fd6b3fff640 1 -- 172.21.15.129:0/3627807432 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd6bc021090 con 0x7fd6cc0ccd60 2024-03-23T15:55:30.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:30.800 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:30.800 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:30.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd6b806b810 msgr2=0x7fd6b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd6b806b810 0x7fd6b806dcd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0036000 tx=0x7fd6c0008030 comp rx=0 tx=0).stop 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 msgr2=0x7fd6cc1f0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 0x7fd6cc1f0cf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00e870 tx=0x7fd6bc00ed30 comp rx=0 tx=0).stop 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 shutdown_connections 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6cc0cef40 0x7fd6cc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd6b806b810 0x7fd6b806dcd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6cc0ceb00 0x7fd6cc1590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 --2- 172.21.15.129:0/3627807432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6cc0ccd60 0x7fd6cc1f0cf0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.799+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 >> 172.21.15.129:0/3627807432 conn(0x7fd6cc0b1840 msgr2=0x7fd6cc0b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.800+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 shutdown_connections 2024-03-23T15:55:30.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:30.800+0000 7fd6d1b9a640 1 -- 172.21.15.129:0/3627807432 wait complete. 2024-03-23T15:55:31.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:31.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:32.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:32.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.519+0000 7fba1a86f640 1 Processor -- start 2024-03-23T15:55:32.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.519+0000 7fba1a86f640 1 -- start start 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.519+0000 7fba1a86f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140ccd60 0x7fba1415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.519+0000 7fba1a86f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fba140cef40 0x7fba1415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.520+0000 7fba1a86f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1415b290 con 0x7fba140ceb00 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.520+0000 7fba1a86f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fba140ce800 con 0x7fba140cef40 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.520+0000 7fba1a86f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fba14154a70 con 0x7fba140ccd60 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.520+0000 7fba137fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fba140ceb00 0x7fba1406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33528/0 (socket says 172.21.15.129:33528) 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.520+0000 7fba137fe640 1 -- 172.21.15.129:0/3267771306 learned_addr learned my addr 172.21.15.129:0/3267771306 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.520+0000 7fba13fff640 1 --2- 172.21.15.129:0/3267771306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140ccd60 0x7fba1415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:32.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.523+0000 7fba13fff640 1 -- 172.21.15.129:0/3267771306 >> v2:172.21.15.129:3300/0 conn(0x7fba140cef40 msgr2=0x7fba1415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:32.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.523+0000 7fba13fff640 1 --2- 172.21.15.129:0/3267771306 >> v2:172.21.15.129:3300/0 conn(0x7fba140cef40 0x7fba1415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:32.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.523+0000 7fba13fff640 1 -- 172.21.15.129:0/3267771306 >> v1:172.21.15.157:6789/0 conn(0x7fba140ceb00 legacy=0x7fba1406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:32.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.523+0000 7fba13fff640 1 -- 172.21.15.129:0/3267771306 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba140cd950 con 0x7fba140ccd60 2024-03-23T15:55:32.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.526+0000 7fba13fff640 1 --2- 172.21.15.129:0/3267771306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140ccd60 0x7fba1415ad50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fba040097a0 tx=0x7fba0402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9432471610eb31ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:32.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.531+0000 7fba12ffd640 1 -- 172.21.15.129:0/3267771306 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04030610 con 0x7fba140ccd60 2024-03-23T15:55:32.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.531+0000 7fba12ffd640 1 -- 172.21.15.129:0/3267771306 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba04030770 con 0x7fba140ccd60 2024-03-23T15:55:32.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.531+0000 7fba12ffd640 1 -- 172.21.15.129:0/3267771306 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04030b00 con 0x7fba140ccd60 2024-03-23T15:55:32.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 -- 172.21.15.129:0/3267771306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140ccd60 msgr2=0x7fba1415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 --2- 172.21.15.129:0/3267771306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140ccd60 0x7fba1415ad50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fba040097a0 tx=0x7fba0402b960 comp rx=0 tx=0).stop 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 -- 172.21.15.129:0/3267771306 shutdown_connections 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 --2- 172.21.15.129:0/3267771306 >> v2:172.21.15.129:3300/0 conn(0x7fba140cef40 0x7fba1415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 --2- 172.21.15.129:0/3267771306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140ccd60 0x7fba1415ad50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 -- 172.21.15.129:0/3267771306 >> 172.21.15.129:0/3267771306 conn(0x7fba140b1840 msgr2=0x7fba140b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 -- 172.21.15.129:0/3267771306 shutdown_connections 2024-03-23T15:55:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.532+0000 7fba1a86f640 1 -- 172.21.15.129:0/3267771306 wait complete. 2024-03-23T15:55:32.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.533+0000 7fba1a86f640 1 Processor -- start 2024-03-23T15:55:32.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.533+0000 7fba1a86f640 1 -- start start 2024-03-23T15:55:32.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba1a86f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 0x7fba140c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:32.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba1a86f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba140ceb00 0x7fba1414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:32.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba1a86f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140cef40 0x7fba140cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:32.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba1a86f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fba140c6dd0 con 0x7fba140ccd60 2024-03-23T15:55:32.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba1a86f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fba140c70b0 con 0x7fba140ceb00 2024-03-23T15:55:32.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba1a86f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fba140c7390 con 0x7fba140cef40 2024-03-23T15:55:32.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba13fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 0x7fba140c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:32.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba18e0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140cef40 0x7fba140cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:32.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba137fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba140ceb00 0x7fba1414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:32.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba13fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 0x7fba140c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40738/0 (socket says 172.21.15.129:40738) 2024-03-23T15:55:32.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.534+0000 7fba13fff640 1 -- 172.21.15.129:0/2279140135 learned_addr learned my addr 172.21.15.129:0/2279140135 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:32.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.538+0000 7fba13fff640 1 -- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140cef40 msgr2=0x7fba140cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:32.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.538+0000 7fba13fff640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140cef40 0x7fba140cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:32.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.538+0000 7fba13fff640 1 -- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba140ceb00 msgr2=0x7fba1414f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:32.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.538+0000 7fba13fff640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba140ceb00 0x7fba1414f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:32.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.538+0000 7fba13fff640 1 -- 172.21.15.129:0/2279140135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba04009400 con 0x7fba140ccd60 2024-03-23T15:55:32.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.540+0000 7fba13fff640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 0x7fba140c8150 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba040098c0 tx=0x7fba04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:32.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.544+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0403e040 con 0x7fba140ccd60 2024-03-23T15:55:32.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.544+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba04039cb0 con 0x7fba140ccd60 2024-03-23T15:55:32.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.544+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba140c7670 con 0x7fba140ccd60 2024-03-23T15:55:32.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.544+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04044630 con 0x7fba140ccd60 2024-03-23T15:55:32.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.544+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba140c7c30 con 0x7fba140ccd60 2024-03-23T15:55:32.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.545+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba1404f530 con 0x7fba140ccd60 2024-03-23T15:55:32.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.568+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fba0404d3b0 con 0x7fba140ccd60 2024-03-23T15:55:32.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.568+0000 7fba10ff9640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fba0006b860 0x7fba0006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:32.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.569+0000 7fba137fe640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fba0006b860 0x7fba0006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:32.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.569+0000 7fba137fe640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fba0006b860 0x7fba0006dd20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fba141f44f0 tx=0x7fba08007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:32.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.813+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fba04039e20 con 0x7fba140ccd60 2024-03-23T15:55:32.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:32.814+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fba04077790 con 0x7fba140ccd60 2024-03-23T15:55:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.042+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba140686f0 con 0x7fba140ccd60 2024-03-23T15:55:33.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.059+0000 7fba10ff9640 1 -- 172.21.15.129:0/2279140135 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fba0407ae50 con 0x7fba140ccd60 2024-03-23T15:55:33.062 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:33.062 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:33.062 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fba0006b860 msgr2=0x7fba0006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fba0006b860 0x7fba0006dd20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fba141f44f0 tx=0x7fba08007e80 comp rx=0 tx=0).stop 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 msgr2=0x7fba140c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 0x7fba140c8150 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba040098c0 tx=0x7fba04004730 comp rx=0 tx=0).stop 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 shutdown_connections 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba140cef40 0x7fba140cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fba0006b860 0x7fba0006dd20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba140ceb00 0x7fba1414f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:33.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 --2- 172.21.15.129:0/2279140135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba140ccd60 0x7fba140c8150 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:33.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 >> 172.21.15.129:0/2279140135 conn(0x7fba140b1840 msgr2=0x7fba14158260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:33.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 shutdown_connections 2024-03-23T15:55:33.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:33.061+0000 7fba1a86f640 1 -- 172.21.15.129:0/2279140135 wait complete. 2024-03-23T15:55:33.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:33.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:34.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:34.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.780+0000 7f2956b25640 1 Processor -- start 2024-03-23T15:55:34.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.780+0000 7f2956b25640 1 -- start start 2024-03-23T15:55:34.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.781+0000 7f2956b25640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f295015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:34.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.781+0000 7f2956b25640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f295015de40 0x7f295015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.781+0000 7f2956b25640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295006c830 con 0x7f295006c030 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.781+0000 7f2956b25640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f29500cc370 con 0x7f295015de40 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.781+0000 7f2956b25640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f295006cae0 con 0x7f295006c410 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.781+0000 7f29550c2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f295015de40 0x7f295015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.782+0000 7f294ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f295015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.782+0000 7f29548c1640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f295006c030 0x7f29500cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:45264/0 (socket says 172.21.15.129:45264) 2024-03-23T15:55:34.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.782+0000 7f29548c1640 1 -- 172.21.15.129:0/2662295817 learned_addr learned my addr 172.21.15.129:0/2662295817 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:34.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.785+0000 7f29550c2640 1 -- 172.21.15.129:0/2662295817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 msgr2=0x7f295015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:34.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.785+0000 7f29550c2640 1 --2- 172.21.15.129:0/2662295817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f295015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:34.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.785+0000 7f29550c2640 1 -- 172.21.15.129:0/2662295817 >> v1:172.21.15.129:6789/0 conn(0x7f295006c030 legacy=0x7f29500cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:34.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.785+0000 7f29550c2640 1 -- 172.21.15.129:0/2662295817 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29500cabc0 con 0x7f295015de40 2024-03-23T15:55:34.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.789+0000 7f29550c2640 1 --2- 172.21.15.129:0/2662295817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f295015e2d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2948009d90 tx=0x7f294802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17be000d16f304c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:34.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.793+0000 7f294f7fe640 1 -- 172.21.15.129:0/2662295817 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294802cc50 con 0x7f295015de40 2024-03-23T15:55:34.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.793+0000 7f294f7fe640 1 -- 172.21.15.129:0/2662295817 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f294802cdb0 con 0x7f295015de40 2024-03-23T15:55:34.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.793+0000 7f294f7fe640 1 -- 172.21.15.129:0/2662295817 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948034660 con 0x7f295015de40 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 -- 172.21.15.129:0/2662295817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 msgr2=0x7f295015e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 --2- 172.21.15.129:0/2662295817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f295015e2d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2948009d90 tx=0x7f294802bea0 comp rx=0 tx=0).stop 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 -- 172.21.15.129:0/2662295817 shutdown_connections 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 --2- 172.21.15.129:0/2662295817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f295015e2d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 --2- 172.21.15.129:0/2662295817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f295015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 -- 172.21.15.129:0/2662295817 >> 172.21.15.129:0/2662295817 conn(0x7f29500cfb20 msgr2=0x7f29500cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:34.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 -- 172.21.15.129:0/2662295817 shutdown_connections 2024-03-23T15:55:34.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.794+0000 7f2956b25640 1 -- 172.21.15.129:0/2662295817 wait complete. 2024-03-23T15:55:34.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.795+0000 7f2956b25640 1 Processor -- start 2024-03-23T15:55:34.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 -- start start 2024-03-23T15:55:34.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f295006c030 0x7f2950155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:34.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f29500cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:34.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f2950156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:34.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29501551a0 con 0x7f295015de40 2024-03-23T15:55:34.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f29501586e0 con 0x7f295006c030 2024-03-23T15:55:34.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.796+0000 7f2956b25640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2950158c10 con 0x7f295006c410 2024-03-23T15:55:34.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.797+0000 7f29550c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f2950156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:34.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.797+0000 7f294ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f29500cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:34.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.797+0000 7f29548c1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f295006c030 0x7f2950155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:34.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.797+0000 7f29550c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f2950156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40764/0 (socket says 172.21.15.129:40764) 2024-03-23T15:55:34.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.797+0000 7f29550c2640 1 -- 172.21.15.129:0/1863357310 learned_addr learned my addr 172.21.15.129:0/1863357310 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:34.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.801+0000 7f29550c2640 1 -- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 msgr2=0x7f29500cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:34.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.801+0000 7f29550c2640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f29500cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:34.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.801+0000 7f29550c2640 1 -- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f295006c030 msgr2=0x7f2950155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:34.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.801+0000 7f29550c2640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f295006c030 0x7f2950155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:34.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.801+0000 7f29550c2640 1 -- 172.21.15.129:0/1863357310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2948009a70 con 0x7f295015de40 2024-03-23T15:55:34.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.803+0000 7f29550c2640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f2950156bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2948009d90 tx=0x7f2948004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.806+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948040650 con 0x7f295015de40 2024-03-23T15:55:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.806+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29480407b0 con 0x7f295015de40 2024-03-23T15:55:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.806+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295015af80 con 0x7f295015de40 2024-03-23T15:55:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.806+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948040b60 con 0x7f295015de40 2024-03-23T15:55:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.807+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2950158930 con 0x7f295015de40 2024-03-23T15:55:34.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:34.807+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2950155350 con 0x7f295015de40 2024-03-23T15:55:35.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.072+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2948040cc0 con 0x7f295015de40 2024-03-23T15:55:35.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.072+0000 7f294d7fa640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f293406b840 0x7f293406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:35.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.072+0000 7f29548c1640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f293406b840 0x7f293406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:35.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.073+0000 7f29548c1640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f293406b840 0x7f293406dd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2944007fd0 tx=0x7f2944007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:35.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.097+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f294807b9b0 con 0x7f295015de40 2024-03-23T15:55:35.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.097+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2948077c50 con 0x7f295015de40 2024-03-23T15:55:35.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.324+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2950155350 con 0x7f295015de40 2024-03-23T15:55:35.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.336+0000 7f294d7fa640 1 -- 172.21.15.129:0/1863357310 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2950155350 con 0x7f295015de40 2024-03-23T15:55:35.339 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:35.339 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:35.340 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f293406b840 msgr2=0x7f293406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f293406b840 0x7f293406dd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2944007fd0 tx=0x7f2944007e80 comp rx=0 tx=0).stop 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 msgr2=0x7f2950156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f2950156bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2948009d90 tx=0x7f2948004a40 comp rx=0 tx=0).stop 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 shutdown_connections 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295015de40 0x7f2950156bc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f295006c410 0x7f29500cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f293406b840 0x7f293406dd00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 --2- 172.21.15.129:0/1863357310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f295006c030 0x7f2950155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 >> 172.21.15.129:0/1863357310 conn(0x7f29500cfb20 msgr2=0x7f29500c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 shutdown_connections 2024-03-23T15:55:35.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:35.339+0000 7f2956b25640 1 -- 172.21.15.129:0/1863357310 wait complete. 2024-03-23T15:55:35.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:35.958 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:36.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:37.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.060+0000 7f130f10c640 1 Processor -- start 2024-03-23T15:55:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.060+0000 7f130f10c640 1 -- start start 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.060+0000 7f130f10c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.060+0000 7f130f10c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f13080c4f70 0x7f13080c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.061+0000 7f130f10c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130815d880 con 0x7f130815ad10 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.061+0000 7f130f10c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f13080c4ae0 con 0x7f13080c4f70 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.061+0000 7f130f10c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f13080cd5e0 con 0x7f1308154270 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.061+0000 7f130cea8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:37.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.062+0000 7f130d6a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f13080c4f70 0x7f13080c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:37.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.062+0000 7f1307fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f130815ad10 0x7f130806c110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33536/0 (socket says 172.21.15.129:33536) 2024-03-23T15:55:37.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.062+0000 7f1307fff640 1 -- 172.21.15.129:0/2619176174 learned_addr learned my addr 172.21.15.129:0/2619176174 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:37.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.065+0000 7f130cea8640 1 -- 172.21.15.129:0/2619176174 >> v2:172.21.15.157:3300/0 conn(0x7f13080c4f70 msgr2=0x7f13080c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:37.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.065+0000 7f130cea8640 1 --2- 172.21.15.129:0/2619176174 >> v2:172.21.15.157:3300/0 conn(0x7f13080c4f70 0x7f13080c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.065+0000 7f130cea8640 1 -- 172.21.15.129:0/2619176174 >> v1:172.21.15.157:6789/0 conn(0x7f130815ad10 legacy=0x7f130806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:37.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.065+0000 7f130cea8640 1 -- 172.21.15.129:0/2619176174 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130815dab0 con 0x7f1308154270 2024-03-23T15:55:37.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.068+0000 7f130cea8640 1 --2- 172.21.15.129:0/2619176174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080ca580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0097a0 tx=0x7f12fc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b33ccb616ba873ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:37.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.073+0000 7f13077fe640 1 -- 172.21.15.129:0/2619176174 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc02f050 con 0x7f1308154270 2024-03-23T15:55:37.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.073+0000 7f13077fe640 1 -- 172.21.15.129:0/2619176174 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12fc031700 con 0x7f1308154270 2024-03-23T15:55:37.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.073+0000 7f13077fe640 1 -- 172.21.15.129:0/2619176174 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc031a90 con 0x7f1308154270 2024-03-23T15:55:37.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 -- 172.21.15.129:0/2619176174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 msgr2=0x7f13080ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 --2- 172.21.15.129:0/2619176174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080ca580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0097a0 tx=0x7f12fc02b7a0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 -- 172.21.15.129:0/2619176174 shutdown_connections 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 --2- 172.21.15.129:0/2619176174 >> v2:172.21.15.157:3300/0 conn(0x7f13080c4f70 0x7f13080c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 --2- 172.21.15.129:0/2619176174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080ca580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 -- 172.21.15.129:0/2619176174 >> 172.21.15.129:0/2619176174 conn(0x7f130806d290 msgr2=0x7f1308072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 -- 172.21.15.129:0/2619176174 shutdown_connections 2024-03-23T15:55:37.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.074+0000 7f130f10c640 1 -- 172.21.15.129:0/2619176174 wait complete. 2024-03-23T15:55:37.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.075+0000 7f130f10c640 1 Processor -- start 2024-03-23T15:55:37.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.075+0000 7f130f10c640 1 -- start start 2024-03-23T15:55:37.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130f10c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13080c4f70 0x7f1308149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:37.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130f10c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:37.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130f10c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 0x7f13080b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:37.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130f10c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13080b3170 con 0x7f130815ad10 2024-03-23T15:55:37.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130f10c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13081f0500 con 0x7f13080c4f70 2024-03-23T15:55:37.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130f10c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f13081f07e0 con 0x7f1308154270 2024-03-23T15:55:37.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f1307fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:37.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130d6a9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 0x7f13080b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:37.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.076+0000 7f130cea8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13080c4f70 0x7f1308149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:37.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.077+0000 7f130d6a9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 0x7f13080b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40778/0 (socket says 172.21.15.129:40778) 2024-03-23T15:55:37.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.077+0000 7f130d6a9640 1 -- 172.21.15.129:0/2710875772 learned_addr learned my addr 172.21.15.129:0/2710875772 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:37.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.080+0000 7f130d6a9640 1 -- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 msgr2=0x7f13080b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:37.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.080+0000 7f130d6a9640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.080+0000 7f130d6a9640 1 -- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13080c4f70 msgr2=0x7f1308149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:37.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.080+0000 7f130d6a9640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13080c4f70 0x7f1308149b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.080+0000 7f130d6a9640 1 -- 172.21.15.129:0/2710875772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc009400 con 0x7f130815ad10 2024-03-23T15:55:37.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.084+0000 7f130d6a9640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 0x7f13080b3e20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f130000efc0 tx=0x7f130000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:37.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.090+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300009ba0 con 0x7f130815ad10 2024-03-23T15:55:37.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.090+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f130000f040 con 0x7f130815ad10 2024-03-23T15:55:37.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.090+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13081f0ac0 con 0x7f130815ad10 2024-03-23T15:55:37.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.090+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13000146c0 con 0x7f130815ad10 2024-03-23T15:55:37.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.090+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13080b33c0 con 0x7f130815ad10 2024-03-23T15:55:37.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.091+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13080b35e0 con 0x7f130815ad10 2024-03-23T15:55:37.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.349+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1300014820 con 0x7f130815ad10 2024-03-23T15:55:37.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.350+0000 7f13057fa640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f12ec06b860 0x7f12ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:37.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.350+0000 7f130cea8640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f12ec06b860 0x7f12ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:37.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.351+0000 7f130cea8640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f12ec06b860 0x7f12ec06dd20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12fc035000 tx=0x7f12fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:37.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.366+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1300059d40 con 0x7f130815ad10 2024-03-23T15:55:37.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.366+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1300055fe0 con 0x7f130815ad10 2024-03-23T15:55:37.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.592+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1308075740 con 0x7f130815ad10 2024-03-23T15:55:37.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.615+0000 7f13057fa640 1 -- 172.21.15.129:0/2710875772 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1300018090 con 0x7f130815ad10 2024-03-23T15:55:37.618 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:37.618 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:37.618 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:37.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f12ec06b860 msgr2=0x7f12ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:37.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f12ec06b860 0x7f12ec06dd20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12fc035000 tx=0x7f12fc008030 comp rx=0 tx=0).stop 2024-03-23T15:55:37.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 msgr2=0x7f13080b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:37.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 0x7f13080b3e20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f130000efc0 tx=0x7f130000c490 comp rx=0 tx=0).stop 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 shutdown_connections 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f130815ad10 0x7f13080b3e20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1308154270 0x7f13080b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f12ec06b860 0x7f12ec06dd20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 --2- 172.21.15.129:0/2710875772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13080c4f70 0x7f1308149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.617+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 >> 172.21.15.129:0/2710875772 conn(0x7f130806d290 msgr2=0x7f1308072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.618+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 shutdown_connections 2024-03-23T15:55:37.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:37.618+0000 7f130f10c640 1 -- 172.21.15.129:0/2710875772 wait complete. 2024-03-23T15:55:38.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:55:38.141 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:55:38.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:38.235 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:39.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:39.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 Processor -- start 2024-03-23T15:55:39.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 -- start start 2024-03-23T15:55:39.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9806cf70 0x7f1a980cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:39.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a9815df80 0x7f1a981540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:39.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a98070640 con 0x7f1a9815dba0 2024-03-23T15:55:39.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f1a980c4c00 con 0x7f1a9815df80 2024-03-23T15:55:39.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.336+0000 7f1a9ddb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9806c240 con 0x7f1a9806cf70 2024-03-23T15:55:39.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.337+0000 7f1a97fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f1a9815df80 0x7f1a981540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:39.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.337+0000 7f1a977fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1a9815dba0 0x7f1a9806ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:33550/0 (socket says 172.21.15.129:33550) 2024-03-23T15:55:39.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.337+0000 7f1a977fe640 1 -- 172.21.15.129:0/956047948 learned_addr learned my addr 172.21.15.129:0/956047948 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:39.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.337+0000 7f1a96ffd640 1 --2- 172.21.15.129:0/956047948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9806cf70 0x7f1a980cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:39.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.339+0000 7f1a97fff640 1 -- 172.21.15.129:0/956047948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9806cf70 msgr2=0x7f1a980cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.339+0000 7f1a97fff640 1 --2- 172.21.15.129:0/956047948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9806cf70 0x7f1a980cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.339+0000 7f1a97fff640 1 -- 172.21.15.129:0/956047948 >> v1:172.21.15.157:6789/0 conn(0x7f1a9815dba0 legacy=0x7f1a9806ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.340+0000 7f1a97fff640 1 -- 172.21.15.129:0/956047948 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a980cbb20 con 0x7f1a9815df80 2024-03-23T15:55:39.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.342+0000 7f1a97fff640 1 --2- 172.21.15.129:0/956047948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a981540a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009d90 tx=0x7f1a9002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d0024d7f5013fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:39.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.345+0000 7f1a967fc640 1 -- 172.21.15.129:0/956047948 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9002cce0 con 0x7f1a9815df80 2024-03-23T15:55:39.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.345+0000 7f1a967fc640 1 -- 172.21.15.129:0/956047948 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9002ce40 con 0x7f1a9815df80 2024-03-23T15:55:39.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.345+0000 7f1a967fc640 1 -- 172.21.15.129:0/956047948 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90034660 con 0x7f1a9815df80 2024-03-23T15:55:39.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/956047948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 msgr2=0x7f1a981540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/956047948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a981540a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009d90 tx=0x7f1a9002bf30 comp rx=0 tx=0).stop 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/956047948 shutdown_connections 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/956047948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a981540a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/956047948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9806cf70 0x7f1a980cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/956047948 >> 172.21.15.129:0/956047948 conn(0x7f1a98071300 msgr2=0x7f1a980716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/956047948 shutdown_connections 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.346+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/956047948 wait complete. 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.347+0000 7f1a9ddb5640 1 Processor -- start 2024-03-23T15:55:39.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.347+0000 7f1a9ddb5640 1 -- start start 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.348+0000 7f1a9ddb5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 0x7f1a980cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.348+0000 7f1a9ddb5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9815dba0 0x7f1a9815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.348+0000 7f1a9ddb5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a9815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.348+0000 7f1a9ddb5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9814ed70 con 0x7f1a9815df80 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.348+0000 7f1a9ddb5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a981e3be0 con 0x7f1a9806cf70 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.348+0000 7f1a9ddb5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a981e3e40 con 0x7f1a9815dba0 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.349+0000 7f1a977fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 0x7f1a980cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.349+0000 7f1a97fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a9815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.349+0000 7f1a96ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9815dba0 0x7f1a9815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.349+0000 7f1a977fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 0x7f1a980cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:36230/0 (socket says 172.21.15.129:36230) 2024-03-23T15:55:39.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.349+0000 7f1a97fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a9815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40786/0 (socket says 172.21.15.129:40786) 2024-03-23T15:55:39.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.349+0000 7f1a977fe640 1 -- 172.21.15.129:0/3158479937 learned_addr learned my addr 172.21.15.129:0/3158479937 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:39.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.354+0000 7f1a977fe640 1 -- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9815dba0 msgr2=0x7f1a9815bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:39.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.354+0000 7f1a977fe640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9815dba0 0x7f1a9815bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.354+0000 7f1a977fe640 1 -- 172.21.15.129:0/3158479937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 msgr2=0x7f1a9815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:39.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.354+0000 7f1a977fe640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a9815c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.354+0000 7f1a977fe640 1 -- 172.21.15.129:0/3158479937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a90009a70 con 0x7f1a9806cf70 2024-03-23T15:55:39.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.357+0000 7f1a977fe640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 0x7f1a980cd950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c400 tx=0x7f1a8800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:39.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.362+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88005ce0 con 0x7f1a9806cf70 2024-03-23T15:55:39.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.362+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88005e40 con 0x7f1a9806cf70 2024-03-23T15:55:39.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.362+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a981471c0 con 0x7f1a9806cf70 2024-03-23T15:55:39.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.362+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a880072d0 con 0x7f1a9806cf70 2024-03-23T15:55:39.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.362+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a98147760 con 0x7f1a9806cf70 2024-03-23T15:55:39.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.363+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a98075740 con 0x7f1a9806cf70 2024-03-23T15:55:39.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.660+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a880074f0 con 0x7f1a9806cf70 2024-03-23T15:55:39.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.661+0000 7f1a7ffff640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b860 0x7f1a8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:39.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.661+0000 7f1a96ffd640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b860 0x7f1a8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:39.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.662+0000 7f1a96ffd640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b860 0x7f1a8406dd20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1a980c9240 tx=0x7f1a8c009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:39.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.665+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1a880583f0 con 0x7f1a9806cf70 2024-03-23T15:55:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.677+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a88054690 con 0x7f1a9806cf70 2024-03-23T15:55:39.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.901+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a980c47a0 con 0x7f1a9806cf70 2024-03-23T15:55:39.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.917+0000 7f1a7ffff640 1 -- 172.21.15.129:0/3158479937 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1a88017070 con 0x7f1a9806cf70 2024-03-23T15:55:39.920 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:39.920 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b860 msgr2=0x7f1a8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b860 0x7f1a8406dd20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1a980c9240 tx=0x7f1a8c009110 comp rx=0 tx=0).stop 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 msgr2=0x7f1a980cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 0x7f1a980cd950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c400 tx=0x7f1a8800c8c0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 shutdown_connections 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a9815df80 0x7f1a9815c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a9815dba0 0x7f1a9815bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1a8406b860 0x7f1a8406dd20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 --2- 172.21.15.129:0/3158479937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a9806cf70 0x7f1a980cd950 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 >> 172.21.15.129:0/3158479937 conn(0x7f1a98071300 msgr2=0x7f1a98149c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 shutdown_connections 2024-03-23T15:55:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:39.920+0000 7f1a9ddb5640 1 -- 172.21.15.129:0/3158479937 wait complete. 2024-03-23T15:55:40.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:40.539 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:41.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.640+0000 7fb606d81640 1 Processor -- start 2024-03-23T15:55:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.640+0000 7fb606d81640 1 -- start start 2024-03-23T15:55:41.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.641+0000 7fb606d81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60015a2e0 0x7fb60006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:41.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.641+0000 7fb606d81640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb60006c0b0 0x7fb6000c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:41.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.641+0000 7fb606d81640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6000cbcb0 con 0x7fb6001544b0 2024-03-23T15:55:41.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.641+0000 7fb606d81640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb6000c5be0 con 0x7fb60006c0b0 2024-03-23T15:55:41.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.641+0000 7fb606d81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb60015b280 con 0x7fb60015a2e0 2024-03-23T15:55:41.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.642+0000 7fb5fffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60015a2e0 0x7fb60006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:41.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.642+0000 7fb604b1d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb6001544b0 0x7fb6000caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60032/0 (socket says 172.21.15.129:60032) 2024-03-23T15:55:41.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.642+0000 7fb604b1d640 1 -- 172.21.15.129:0/2492023165 learned_addr learned my addr 172.21.15.129:0/2492023165 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:41.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.642+0000 7fb60531e640 1 --2- 172.21.15.129:0/2492023165 >> v2:172.21.15.157:3300/0 conn(0x7fb60006c0b0 0x7fb6000c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:41.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.645+0000 7fb5fffff640 1 -- 172.21.15.129:0/2492023165 >> v2:172.21.15.157:3300/0 conn(0x7fb60006c0b0 msgr2=0x7fb6000c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:41.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.645+0000 7fb5fffff640 1 --2- 172.21.15.129:0/2492023165 >> v2:172.21.15.157:3300/0 conn(0x7fb60006c0b0 0x7fb6000c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:41.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.645+0000 7fb5fffff640 1 -- 172.21.15.129:0/2492023165 >> v1:172.21.15.157:6789/0 conn(0x7fb6001544b0 legacy=0x7fb6000caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:41.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.645+0000 7fb5fffff640 1 -- 172.21.15.129:0/2492023165 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60015a9e0 con 0x7fb60015a2e0 2024-03-23T15:55:41.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.647+0000 7fb5fffff640 1 --2- 172.21.15.129:0/2492023165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60015a2e0 0x7fb60006bb70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0097a0 tx=0x7fb5ec02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0db5a639eac71fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:41.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.650+0000 7fb5ff7fe640 1 -- 172.21.15.129:0/2492023165 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec030610 con 0x7fb60015a2e0 2024-03-23T15:55:41.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.652+0000 7fb5ff7fe640 1 -- 172.21.15.129:0/2492023165 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ec030770 con 0x7fb60015a2e0 2024-03-23T15:55:41.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb5ff7fe640 1 -- 172.21.15.129:0/2492023165 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec030b00 con 0x7fb60015a2e0 2024-03-23T15:55:41.656 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:01.498 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:41.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 -- 172.21.15.129:0/2492023165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60015a2e0 msgr2=0x7fb60006bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 --2- 172.21.15.129:0/2492023165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60015a2e0 0x7fb60006bb70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0097a0 tx=0x7fb5ec02b960 comp rx=0 tx=0).stop 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 -- 172.21.15.129:0/2492023165 shutdown_connections 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 --2- 172.21.15.129:0/2492023165 >> v2:172.21.15.157:3300/0 conn(0x7fb60006c0b0 0x7fb6000c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 --2- 172.21.15.129:0/2492023165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60015a2e0 0x7fb60006bb70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 -- 172.21.15.129:0/2492023165 >> 172.21.15.129:0/2492023165 conn(0x7fb600061b50 msgr2=0x7fb600061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 -- 172.21.15.129:0/2492023165 shutdown_connections 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.653+0000 7fb606d81640 1 -- 172.21.15.129:0/2492023165 wait complete. 2024-03-23T15:55:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.655+0000 7fb606d81640 1 Processor -- start 2024-03-23T15:55:41.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.655+0000 7fb606d81640 1 -- start start 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb606d81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 0x7fb6001560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb606d81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6001544b0 0x7fb6000c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb606d81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb60015a2e0 0x7fb600157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb606d81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6001579b0 con 0x7fb60015a2e0 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb606d81640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb6000cda40 con 0x7fb6001544b0 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb606d81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb6000cdd20 con 0x7fb60006c0b0 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb604b1d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 0x7fb6001560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.656+0000 7fb5fffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6001544b0 0x7fb6000c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.657+0000 7fb604b1d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 0x7fb6001560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:41044/0 (socket says 172.21.15.129:41044) 2024-03-23T15:55:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.657+0000 7fb604b1d640 1 -- 172.21.15.129:0/468310769 learned_addr learned my addr 172.21.15.129:0/468310769 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.661+0000 7fb604b1d640 1 -- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6001544b0 msgr2=0x7fb6000c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.661+0000 7fb604b1d640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6001544b0 0x7fb6000c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.661+0000 7fb604b1d640 1 -- 172.21.15.129:0/468310769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb60015a2e0 msgr2=0x7fb600157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.661+0000 7fb604b1d640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb60015a2e0 0x7fb600157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.661+0000 7fb604b1d640 1 -- 172.21.15.129:0/468310769 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ec009400 con 0x7fb60006c0b0 2024-03-23T15:55:41.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.664+0000 7fb604b1d640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 0x7fb6001560c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c4d0 tx=0x7fb5f400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:41.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.669+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4005ce0 con 0x7fb60006c0b0 2024-03-23T15:55:41.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.669+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f4005e40 con 0x7fb60006c0b0 2024-03-23T15:55:41.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.669+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4007270 con 0x7fb60006c0b0 2024-03-23T15:55:41.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.669+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6000ce000 con 0x7fb60006c0b0 2024-03-23T15:55:41.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.669+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6001558c0 con 0x7fb60006c0b0 2024-03-23T15:55:41.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.670+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6000c6e80 con 0x7fb60006c0b0 2024-03-23T15:55:41.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.694+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb5f40073d0 con 0x7fb60006c0b0 2024-03-23T15:55:41.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.694+0000 7fb5fd7fa640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb5e406b890 0x7fb5e406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:41.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.694+0000 7fb5fffff640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb5e406b890 0x7fb5e406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:41.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.695+0000 7fb5fffff640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb5e406b890 0x7fb5e406dd50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec036000 tx=0x7fb5ec03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:41.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:41.700+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb5f4058330 con 0x7fb60006c0b0 2024-03-23T15:55:41.791 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:01.633 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:41.917 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:01.715 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.004 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:01.802 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.007+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb5f40545d0 con 0x7fb60006c0b0 2024-03-23T15:55:42.178 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:02.021 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.192 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:09:01.990 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.268+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb600075740 con 0x7fb60006c0b0 2024-03-23T15:55:42.282 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:09:02.080 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.449 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:02.247 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.730 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:09:02.528 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:55:42.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.839+0000 7fb5fd7fa640 1 -- 172.21.15.129:0/468310769 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb5f4017020 con 0x7fb60006c0b0 2024-03-23T15:55:42.842 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:42.843 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:42.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb5e406b890 msgr2=0x7fb5e406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb5e406b890 0x7fb5e406dd50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec036000 tx=0x7fb5ec03b040 comp rx=0 tx=0).stop 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 msgr2=0x7fb6001560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 0x7fb6001560c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c4d0 tx=0x7fb5f400c990 comp rx=0 tx=0).stop 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 shutdown_connections 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb60015a2e0 0x7fb600157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb5e406b890 0x7fb5e406dd50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6001544b0 0x7fb6000c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 --2- 172.21.15.129:0/468310769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb60006c0b0 0x7fb6001560c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 >> 172.21.15.129:0/468310769 conn(0x7fb600061b50 msgr2=0x7fb6000c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 shutdown_connections 2024-03-23T15:55:42.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:42.842+0000 7fb606d81640 1 -- 172.21.15.129:0/468310769 wait complete. 2024-03-23T15:55:43.460 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:43.460 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:44.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.561+0000 7f84280ed640 1 Processor -- start 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.561+0000 7f84280ed640 1 -- start start 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.562+0000 7f84280ed640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f842006bd40 0x7f84200cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.562+0000 7f84280ed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842015e090 con 0x7f8420154340 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.562+0000 7f84280ed640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842006ca10 con 0x7f842015e520 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.562+0000 7f84280ed640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f842015d860 con 0x7f842006bd40 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.563+0000 7f8425688640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f842015e520 0x7f842015aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60048/0 (socket says 172.21.15.129:60048) 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.563+0000 7f8425688640 1 -- 172.21.15.129:0/489707309 learned_addr learned my addr 172.21.15.129:0/489707309 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.563+0000 7f842668a640 1 --2- 172.21.15.129:0/489707309 >> v2:172.21.15.129:3300/0 conn(0x7f842006bd40 0x7f84200cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.566+0000 7f842668a640 1 -- 172.21.15.129:0/489707309 >> v1:172.21.15.157:6789/0 conn(0x7f842015e520 legacy=0x7f842015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.566+0000 7f842668a640 1 -- 172.21.15.129:0/489707309 >> v1:172.21.15.129:6789/0 conn(0x7f8420154340 legacy=0x7f84200c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.566+0000 7f842668a640 1 -- 172.21.15.129:0/489707309 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8420154720 con 0x7f842006bd40 2024-03-23T15:55:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.571+0000 7f842668a640 1 --2- 172.21.15.129:0/489707309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f842006bd40 0x7f84200cc160 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8418009d90 tx=0x7f841802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b28d5f43f62b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:44.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.575+0000 7f8424e87640 1 -- 172.21.15.129:0/489707309 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841802cd80 con 0x7f842006bd40 2024-03-23T15:55:44.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.575+0000 7f8424e87640 1 -- 172.21.15.129:0/489707309 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8418034440 con 0x7f842006bd40 2024-03-23T15:55:44.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.575+0000 7f8424e87640 1 -- 172.21.15.129:0/489707309 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180347d0 con 0x7f842006bd40 2024-03-23T15:55:44.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 -- 172.21.15.129:0/489707309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f842006bd40 msgr2=0x7f84200cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:44.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 --2- 172.21.15.129:0/489707309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f842006bd40 0x7f84200cc160 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8418009d90 tx=0x7f841802bfd0 comp rx=0 tx=0).stop 2024-03-23T15:55:44.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 -- 172.21.15.129:0/489707309 shutdown_connections 2024-03-23T15:55:44.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 --2- 172.21.15.129:0/489707309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f842006bd40 0x7f84200cc160 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:44.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 -- 172.21.15.129:0/489707309 >> 172.21.15.129:0/489707309 conn(0x7f8420071dc0 msgr2=0x7f84200721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:44.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 -- 172.21.15.129:0/489707309 shutdown_connections 2024-03-23T15:55:44.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.576+0000 7f84280ed640 1 -- 172.21.15.129:0/489707309 wait complete. 2024-03-23T15:55:44.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.578+0000 7f84280ed640 1 Processor -- start 2024-03-23T15:55:44.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.578+0000 7f84280ed640 1 -- start start 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f84280ed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f842006bd40 0x7f842014f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f84280ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 0x7f84200c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f84280ed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f842015e520 0x7f8420149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f84280ed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f84201f3c00 con 0x7f8420154340 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f84280ed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f84201f0430 con 0x7f842006bd40 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f84280ed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f84201f40f0 con 0x7f842015e520 2024-03-23T15:55:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.579+0000 7f8425688640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 0x7f84200c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:44.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.580+0000 7f8425688640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 0x7f84200c4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47172/0 (socket says 172.21.15.129:47172) 2024-03-23T15:55:44.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.580+0000 7f8425688640 1 -- 172.21.15.129:0/3927329145 learned_addr learned my addr 172.21.15.129:0/3927329145 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:44.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.580+0000 7f8425e89640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f842006bd40 0x7f842014f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:44.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.580+0000 7f842668a640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f842015e520 0x7f8420149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:44.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.583+0000 7f8425688640 1 -- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f842015e520 msgr2=0x7f8420149250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:44.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.583+0000 7f8425688640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f842015e520 0x7f8420149250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:44.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.583+0000 7f8425688640 1 -- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f842006bd40 msgr2=0x7f842014f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:44.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.583+0000 7f8425688640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f842006bd40 0x7f842014f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:44.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.583+0000 7f8425688640 1 -- 172.21.15.129:0/3927329145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8418009a70 con 0x7f8420154340 2024-03-23T15:55:44.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.585+0000 7f8425688640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 0x7f84200c4a80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f841000c430 tx=0x7f841000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:44.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.588+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410005d30 con 0x7f8420154340 2024-03-23T15:55:44.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.588+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8410005e90 con 0x7f8420154340 2024-03-23T15:55:44.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.588+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842014f320 con 0x7f8420154340 2024-03-23T15:55:44.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.588+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84100072d0 con 0x7f8420154340 2024-03-23T15:55:44.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.588+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84201f3e10 con 0x7f8420154340 2024-03-23T15:55:44.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.589+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84201efc80 con 0x7f8420154340 2024-03-23T15:55:44.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.608+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8410007430 con 0x7f8420154340 2024-03-23T15:55:44.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.608+0000 7f84167fc640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f840406b840 0x7f840406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:44.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.609+0000 7f8425e89640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f840406b840 0x7f840406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:44.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.609+0000 7f8425e89640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f840406b840 0x7f840406dd00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f841c001780 tx=0x7f841c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.851+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f84100581e0 con 0x7f8420154340 2024-03-23T15:55:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:44.852+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8410054480 con 0x7f8420154340 2024-03-23T15:55:45.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.077+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84201efc80 con 0x7f8420154340 2024-03-23T15:55:45.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.099+0000 7f84167fc640 1 -- 172.21.15.129:0/3927329145 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f84201efc80 con 0x7f8420154340 2024-03-23T15:55:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:45.102 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:45.103 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f840406b840 msgr2=0x7f840406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f840406b840 0x7f840406dd00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f841c001780 tx=0x7f841c009040 comp rx=0 tx=0).stop 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 msgr2=0x7f84200c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 0x7f84200c4a80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f841000c430 tx=0x7f841000c8f0 comp rx=0 tx=0).stop 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 shutdown_connections 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f842015e520 0x7f8420149250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8420154340 0x7f84200c4a80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f840406b840 0x7f840406dd00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 --2- 172.21.15.129:0/3927329145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f842006bd40 0x7f842014f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 >> 172.21.15.129:0/3927329145 conn(0x7f8420071dc0 msgr2=0x7f8420152320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 shutdown_connections 2024-03-23T15:55:45.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:45.102+0000 7f84280ed640 1 -- 172.21.15.129:0/3927329145 wait complete. 2024-03-23T15:55:45.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:45.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:46.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:46.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 Processor -- start 2024-03-23T15:55:46.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 -- start start 2024-03-23T15:55:46.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0fc80ccd60 0x7f0fc815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:46.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0fc80ceb00 0x7f0fc806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:46.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc815b290 con 0x7f0fc80cef40 2024-03-23T15:55:46.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc80ce800 con 0x7f0fc80ccd60 2024-03-23T15:55:46.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.821+0000 7f0fcd15e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc8154a70 con 0x7f0fc80ceb00 2024-03-23T15:55:46.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.822+0000 7f0fc659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0fc80ceb00 0x7f0fc806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:46.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.822+0000 7f0fc6d9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0fc80ccd60 0x7f0fc815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:46.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.822+0000 7f0fc659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0fc80ceb00 0x7f0fc806c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38038/0 (socket says 172.21.15.129:38038) 2024-03-23T15:55:46.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.822+0000 7f0fc759e640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f0fc80cef40 0x7f0fc815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43580/0 (socket says 172.21.15.129:43580) 2024-03-23T15:55:46.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.822+0000 7f0fc659c640 1 -- 172.21.15.129:0/4009849177 learned_addr learned my addr 172.21.15.129:0/4009849177 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:46.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.825+0000 7f0fc659c640 1 -- 172.21.15.129:0/4009849177 >> v2:172.21.15.129:3300/0 conn(0x7f0fc80ccd60 msgr2=0x7f0fc815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:46.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.825+0000 7f0fc659c640 1 --2- 172.21.15.129:0/4009849177 >> v2:172.21.15.129:3300/0 conn(0x7f0fc80ccd60 0x7f0fc815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:46.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.825+0000 7f0fc659c640 1 -- 172.21.15.129:0/4009849177 >> v1:172.21.15.129:6789/0 conn(0x7f0fc80cef40 legacy=0x7f0fc815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:46.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.825+0000 7f0fc659c640 1 -- 172.21.15.129:0/4009849177 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc80cd950 con 0x7f0fc80ceb00 2024-03-23T15:55:46.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.827+0000 7f0fc659c640 1 --2- 172.21.15.129:0/4009849177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ceb00 0x7f0fc806c270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fc808d1a0 tx=0x7f0fbc02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d73c8a543251034 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fc5d9b640 1 -- 172.21.15.129:0/4009849177 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc02f850 con 0x7f0fc80ceb00 2024-03-23T15:55:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fc5d9b640 1 -- 172.21.15.129:0/4009849177 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fbc02f9b0 con 0x7f0fc80ceb00 2024-03-23T15:55:46.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fc5d9b640 1 -- 172.21.15.129:0/4009849177 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc02fd40 con 0x7f0fc80ceb00 2024-03-23T15:55:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 -- 172.21.15.129:0/4009849177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ceb00 msgr2=0x7f0fc806c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/4009849177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ceb00 0x7f0fc806c270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fc808d1a0 tx=0x7f0fbc02ba30 comp rx=0 tx=0).stop 2024-03-23T15:55:46.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 -- 172.21.15.129:0/4009849177 shutdown_connections 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/4009849177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ceb00 0x7f0fc806c270 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/4009849177 >> v2:172.21.15.129:3300/0 conn(0x7f0fc80ccd60 0x7f0fc815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 -- 172.21.15.129:0/4009849177 >> 172.21.15.129:0/4009849177 conn(0x7f0fc80b1840 msgr2=0x7f0fc80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 -- 172.21.15.129:0/4009849177 shutdown_connections 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.832+0000 7f0fcd15e640 1 -- 172.21.15.129:0/4009849177 wait complete. 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.834+0000 7f0fcd15e640 1 Processor -- start 2024-03-23T15:55:46.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.834+0000 7f0fcd15e640 1 -- start start 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fcd15e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ccd60 0x7f0fc815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fcd15e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fc80ceb00 0x7f0fc80cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fcd15e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc80cef40 0x7f0fc81f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fcd15e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc80c70b0 con 0x7f0fc80cef40 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fcd15e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81f1be0 con 0x7f0fc80ccd60 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fcd15e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc81f15d0 con 0x7f0fc80ceb00 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fc6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ccd60 0x7f0fc815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fc759e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc80cef40 0x7f0fc81f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fc6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ccd60 0x7f0fc815c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38048/0 (socket says 172.21.15.129:38048) 2024-03-23T15:55:46.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fc6d9d640 1 -- 172.21.15.129:0/2830711006 learned_addr learned my addr 172.21.15.129:0/2830711006 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:46.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.835+0000 7f0fc659c640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fc80ceb00 0x7f0fc80cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:46.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.839+0000 7f0fc759e640 1 -- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fc80ceb00 msgr2=0x7f0fc80cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:46.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.839+0000 7f0fc759e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fc80ceb00 0x7f0fc80cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:46.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.839+0000 7f0fc759e640 1 -- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ccd60 msgr2=0x7f0fc815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:46.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.839+0000 7f0fc759e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ccd60 0x7f0fc815c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.839+0000 7f0fc759e640 1 -- 172.21.15.129:0/2830711006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fbc0094d0 con 0x7f0fc80cef40 2024-03-23T15:55:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.841+0000 7f0fc759e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc80cef40 0x7f0fc81f2070 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c6c0 tx=0x7f0fc000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:46.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.844+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0004d60 con 0x7f0fc80cef40 2024-03-23T15:55:46.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.844+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc000a640 con 0x7f0fc80cef40 2024-03-23T15:55:46.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.844+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc815cfe0 con 0x7f0fc80cef40 2024-03-23T15:55:46.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.844+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc000a9f0 con 0x7f0fc80cef40 2024-03-23T15:55:46.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.845+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc815cd00 con 0x7f0fc80cef40 2024-03-23T15:55:46.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.846+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc804f530 con 0x7f0fc80cef40 2024-03-23T15:55:46.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.869+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0fc000ab50 con 0x7f0fc80cef40 2024-03-23T15:55:46.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.869+0000 7f0faf7fe640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb406b810 0x7f0fb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:46.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.870+0000 7f0fc6d9d640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb406b810 0x7f0fb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:46.870+0000 7f0fc6d9d640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb406b810 0x7f0fb406dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009b60 tx=0x7f0fb8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:47.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.130+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0fc0058270 con 0x7f0fc80cef40 2024-03-23T15:55:47.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.130+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0fc0054510 con 0x7f0fc80cef40 2024-03-23T15:55:47.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.377+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fc806db50 con 0x7f0fc80cef40 2024-03-23T15:55:47.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.391+0000 7f0faf7fe640 1 -- 172.21.15.129:0/2830711006 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0fc0018090 con 0x7f0fc80cef40 2024-03-23T15:55:47.395 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:47.395 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:47.395 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb406b810 msgr2=0x7f0fb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb406b810 0x7f0fb406dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009b60 tx=0x7f0fb8009190 comp rx=0 tx=0).stop 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc80cef40 msgr2=0x7f0fc81f2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc80cef40 0x7f0fc81f2070 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c6c0 tx=0x7f0fc000cb80 comp rx=0 tx=0).stop 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 shutdown_connections 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc80cef40 0x7f0fc81f2070 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0fc80ceb00 0x7f0fc80cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0fb406b810 0x7f0fb406dcd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 --2- 172.21.15.129:0/2830711006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc80ccd60 0x7f0fc815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:47.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 >> 172.21.15.129:0/2830711006 conn(0x7f0fc80b1840 msgr2=0x7f0fc81e5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:47.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 shutdown_connections 2024-03-23T15:55:47.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:47.394+0000 7f0fcd15e640 1 -- 172.21.15.129:0/2830711006 wait complete. 2024-03-23T15:55:48.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:48.012 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:49.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.112+0000 7f25326a4640 1 Processor -- start 2024-03-23T15:55:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.112+0000 7f25326a4640 1 -- start start 2024-03-23T15:55:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.113+0000 7f25326a4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f252c1544d0 0x7f252c0c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.113+0000 7f25326a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c15e470 0x7f252c15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.113+0000 7f25326a4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c15e8b0 con 0x7f252c06b930 2024-03-23T15:55:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.113+0000 7f25326a4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f252c072a70 con 0x7f252c1544d0 2024-03-23T15:55:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.113+0000 7f25326a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f252c15b2c0 con 0x7f252c15e470 2024-03-23T15:55:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.114+0000 7f2530c41640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f252c06b930 0x7f252c0ccb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:43596/0 (socket says 172.21.15.129:43596) 2024-03-23T15:55:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.114+0000 7f2530c41640 1 -- 172.21.15.129:0/3061089658 learned_addr learned my addr 172.21.15.129:0/3061089658 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.114+0000 7f252b7fe640 1 --2- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c15e470 0x7f252c15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.114+0000 7f252bfff640 1 --2- 172.21.15.129:0/3061089658 >> v2:172.21.15.157:3300/0 conn(0x7f252c1544d0 0x7f252c0c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.117+0000 7f252affd640 1 -- 172.21.15.129:0/3061089658 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1144872872 0 0) 0x7f252c15e8b0 con 0x7f252c06b930 2024-03-23T15:55:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.117+0000 7f252affd640 1 -- 172.21.15.129:0/3061089658 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2514003520 con 0x7f252c06b930 2024-03-23T15:55:49.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.118+0000 7f252bfff640 1 -- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c15e470 msgr2=0x7f252c15ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:49.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.118+0000 7f252bfff640 1 --2- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c15e470 0x7f252c15ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.118+0000 7f252bfff640 1 -- 172.21.15.129:0/3061089658 >> v1:172.21.15.129:6789/0 conn(0x7f252c06b930 legacy=0x7f252c0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:49.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.118+0000 7f252bfff640 1 -- 172.21.15.129:0/3061089658 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c072ce0 con 0x7f252c1544d0 2024-03-23T15:55:49.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.120+0000 7f252bfff640 1 --2- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c1544d0 0x7f252c0c7ca0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f251c0097a0 tx=0x7f251c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16715f75809f1123 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.125+0000 7f252affd640 1 -- 172.21.15.129:0/3061089658 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c031830 con 0x7f252c1544d0 2024-03-23T15:55:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.125+0000 7f252affd640 1 -- 172.21.15.129:0/3061089658 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c031990 con 0x7f252c1544d0 2024-03-23T15:55:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.125+0000 7f252affd640 1 -- 172.21.15.129:0/3061089658 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c031d20 con 0x7f252c1544d0 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 -- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c1544d0 msgr2=0x7f252c0c7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 --2- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c1544d0 0x7f252c0c7ca0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f251c0097a0 tx=0x7f251c02b8d0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 -- 172.21.15.129:0/3061089658 shutdown_connections 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 --2- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c15e470 0x7f252c15ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 --2- 172.21.15.129:0/3061089658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c1544d0 0x7f252c0c7ca0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 -- 172.21.15.129:0/3061089658 >> 172.21.15.129:0/3061089658 conn(0x7f252c06e230 msgr2=0x7f252c06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 -- 172.21.15.129:0/3061089658 shutdown_connections 2024-03-23T15:55:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.126+0000 7f25326a4640 1 -- 172.21.15.129:0/3061089658 wait complete. 2024-03-23T15:55:49.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.127+0000 7f25326a4640 1 Processor -- start 2024-03-23T15:55:49.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.127+0000 7f25326a4640 1 -- start start 2024-03-23T15:55:49.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.128+0000 7f25326a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c06b930 0x7f252c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.128+0000 7f25326a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c1544d0 0x7f252c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.128+0000 7f25326a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 0x7f252c155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.128+0000 7f25326a4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f252c1570e0 con 0x7f252c15e470 2024-03-23T15:55:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.128+0000 7f25326a4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f252c159aa0 con 0x7f252c06b930 2024-03-23T15:55:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.128+0000 7f25326a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f252c157c90 con 0x7f252c1544d0 2024-03-23T15:55:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.129+0000 7f252bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c06b930 0x7f252c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:49.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.129+0000 7f2530c41640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 0x7f252c155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:49.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.129+0000 7f252b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c1544d0 0x7f252c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:49.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.129+0000 7f2530c41640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 0x7f252c155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47204/0 (socket says 172.21.15.129:47204) 2024-03-23T15:55:49.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.129+0000 7f2530c41640 1 -- 172.21.15.129:0/1696004058 learned_addr learned my addr 172.21.15.129:0/1696004058 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.132+0000 7f2530c41640 1 -- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c1544d0 msgr2=0x7f252c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.132+0000 7f2530c41640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c1544d0 0x7f252c155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.132+0000 7f2530c41640 1 -- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c06b930 msgr2=0x7f252c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.132+0000 7f2530c41640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c06b930 0x7f252c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.132+0000 7f2530c41640 1 -- 172.21.15.129:0/1696004058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c009400 con 0x7f252c15e470 2024-03-23T15:55:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.134+0000 7f2530c41640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 0x7f252c155990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f252400caa0 tx=0x7f252400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:49.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.138+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252400ac00 con 0x7f252c15e470 2024-03-23T15:55:49.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.138+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252400ad60 con 0x7f252c15e470 2024-03-23T15:55:49.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.138+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252c154fc0 con 0x7f252c15e470 2024-03-23T15:55:49.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.138+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524012620 con 0x7f252c15e470 2024-03-23T15:55:49.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.138+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f252c159f90 con 0x7f252c15e470 2024-03-23T15:55:49.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.139+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252c155150 con 0x7f252c15e470 2024-03-23T15:55:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.406+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2524007070 con 0x7f252c15e470 2024-03-23T15:55:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.406+0000 7f2528ff9640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f251806b890 0x7f251806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.407+0000 7f252bfff640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f251806b890 0x7f251806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.407+0000 7f252bfff640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f251806b890 0x7f251806dd50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f251c036000 tx=0x7f251c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:49.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.428+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2524058d20 con 0x7f252c15e470 2024-03-23T15:55:49.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.428+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2524054fc0 con 0x7f252c15e470 2024-03-23T15:55:49.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.661+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f252c075740 con 0x7f252c15e470 2024-03-23T15:55:49.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.681+0000 7f2528ff9640 1 -- 172.21.15.129:0/1696004058 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2524058680 con 0x7f252c15e470 2024-03-23T15:55:49.684 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:49.684 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:49.685 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:49.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f251806b890 msgr2=0x7f251806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f251806b890 0x7f251806dd50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f251c036000 tx=0x7f251c03c040 comp rx=0 tx=0).stop 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 msgr2=0x7f252c155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 0x7f252c155990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f252400caa0 tx=0x7f252400cf60 comp rx=0 tx=0).stop 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 shutdown_connections 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f252c15e470 0x7f252c155990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f252c1544d0 0x7f252c155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f251806b890 0x7f251806dd50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 --2- 172.21.15.129:0/1696004058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f252c06b930 0x7f252c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 >> 172.21.15.129:0/1696004058 conn(0x7f252c06e230 msgr2=0x7f252c152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 shutdown_connections 2024-03-23T15:55:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:49.684+0000 7f25326a4640 1 -- 172.21.15.129:0/1696004058 wait complete. 2024-03-23T15:55:50.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:50.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:51.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:51.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.402+0000 7ffa4b847640 1 Processor -- start 2024-03-23T15:55:51.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.403+0000 7ffa4b847640 1 -- start start 2024-03-23T15:55:51.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.404+0000 7ffa4b847640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffa440c8c80 0x7ffa440c7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:51.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.404+0000 7ffa4b847640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa440c76b0 con 0x7ffa440ccdd0 2024-03-23T15:55:51.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.404+0000 7ffa4b847640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa440c78f0 con 0x7ffa440cc9f0 2024-03-23T15:55:51.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.404+0000 7ffa4b847640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffa440c7fd0 con 0x7ffa440c8c80 2024-03-23T15:55:51.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.404+0000 7ffa48de2640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ffa440ccdd0 0x7ffa44150d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48844/0 (socket says 172.21.15.129:48844) 2024-03-23T15:55:51.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.405+0000 7ffa48de2640 1 -- 172.21.15.129:0/4294500664 learned_addr learned my addr 172.21.15.129:0/4294500664 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:51.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.405+0000 7ffa49de4640 1 --2- 172.21.15.129:0/4294500664 >> v2:172.21.15.157:3300/0 conn(0x7ffa440c8c80 0x7ffa440c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:51.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.407+0000 7ffa3bfff640 1 -- 172.21.15.129:0/4294500664 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970257898 0 0) 0x7ffa34001110 con 0x7ffa440ccdd0 2024-03-23T15:55:51.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.407+0000 7ffa3bfff640 1 -- 172.21.15.129:0/4294500664 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa30005730 con 0x7ffa440ccdd0 2024-03-23T15:55:51.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.410+0000 7ffa49de4640 1 -- 172.21.15.129:0/4294500664 >> v1:172.21.15.157:6789/0 conn(0x7ffa440cc9f0 legacy=0x7ffa440c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.410+0000 7ffa49de4640 1 -- 172.21.15.129:0/4294500664 >> v1:172.21.15.129:6789/0 conn(0x7ffa440ccdd0 legacy=0x7ffa44150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.410+0000 7ffa49de4640 1 -- 172.21.15.129:0/4294500664 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa44072c90 con 0x7ffa440c8c80 2024-03-23T15:55:51.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.417+0000 7ffa49de4640 1 --2- 172.21.15.129:0/4294500664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440c8c80 0x7ffa440c7170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009d90 tx=0x7ffa3c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640410137d684ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.421+0000 7ffa3bfff640 1 -- 172.21.15.129:0/4294500664 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c02cd80 con 0x7ffa440c8c80 2024-03-23T15:55:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.421+0000 7ffa3bfff640 1 -- 172.21.15.129:0/4294500664 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa3c034440 con 0x7ffa440c8c80 2024-03-23T15:55:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.421+0000 7ffa3bfff640 1 -- 172.21.15.129:0/4294500664 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0347d0 con 0x7ffa440c8c80 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 -- 172.21.15.129:0/4294500664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440c8c80 msgr2=0x7ffa440c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 --2- 172.21.15.129:0/4294500664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440c8c80 0x7ffa440c7170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009d90 tx=0x7ffa3c02bfd0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 -- 172.21.15.129:0/4294500664 shutdown_connections 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 --2- 172.21.15.129:0/4294500664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440c8c80 0x7ffa440c7170 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 -- 172.21.15.129:0/4294500664 >> 172.21.15.129:0/4294500664 conn(0x7ffa4406eb20 msgr2=0x7ffa4406c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 -- 172.21.15.129:0/4294500664 shutdown_connections 2024-03-23T15:55:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.422+0000 7ffa4b847640 1 -- 172.21.15.129:0/4294500664 wait complete. 2024-03-23T15:55:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.423+0000 7ffa4b847640 1 Processor -- start 2024-03-23T15:55:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 -- start start 2024-03-23T15:55:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa440c8c80 0x7ffa44148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440cc9f0 0x7ffa44158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 0x7ffa44159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4415a410 con 0x7ffa440ccdd0 2024-03-23T15:55:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441e3240 con 0x7ffa440cc9f0 2024-03-23T15:55:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.424+0000 7ffa4b847640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa441eca40 con 0x7ffa440c8c80 2024-03-23T15:55:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.425+0000 7ffa49de4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 0x7ffa44159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.425+0000 7ffa48de2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440cc9f0 0x7ffa44158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.425+0000 7ffa495e3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa440c8c80 0x7ffa44148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.425+0000 7ffa49de4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 0x7ffa44159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47086/0 (socket says 172.21.15.129:47086) 2024-03-23T15:55:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.425+0000 7ffa49de4640 1 -- 172.21.15.129:0/3354086985 learned_addr learned my addr 172.21.15.129:0/3354086985 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.430+0000 7ffa49de4640 1 -- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa440c8c80 msgr2=0x7ffa44148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.430+0000 7ffa49de4640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa440c8c80 0x7ffa44148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.430+0000 7ffa49de4640 1 -- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440cc9f0 msgr2=0x7ffa44158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.430+0000 7ffa49de4640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440cc9f0 0x7ffa44158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.430+0000 7ffa49de4640 1 -- 172.21.15.129:0/3354086985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c009a70 con 0x7ffa440ccdd0 2024-03-23T15:55:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.433+0000 7ffa49de4640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 0x7ffa44159420 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c02c5e0 tx=0x7ffa3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.440+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c03e810 con 0x7ffa440ccdd0 2024-03-23T15:55:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.440+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa3c03e970 con 0x7ffa440ccdd0 2024-03-23T15:55:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.440+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa44158a50 con 0x7ffa440ccdd0 2024-03-23T15:55:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.440+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c03ed20 con 0x7ffa440ccdd0 2024-03-23T15:55:51.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.441+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa441e2f60 con 0x7ffa440ccdd0 2024-03-23T15:55:51.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.442+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa44158be0 con 0x7ffa440ccdd0 2024-03-23T15:55:51.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.465+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffa3c051480 con 0x7ffa440ccdd0 2024-03-23T15:55:51.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.466+0000 7ffa39ffb640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2806b840 0x7ffa2806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:51.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.466+0000 7ffa495e3640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2806b840 0x7ffa2806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:51.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.467+0000 7ffa495e3640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2806b840 0x7ffa2806dd00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa40000f60 tx=0x7ffa40009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:51.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.729+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ffa3c07bfb0 con 0x7ffa440ccdd0 2024-03-23T15:55:51.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.729+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa3c078250 con 0x7ffa440ccdd0 2024-03-23T15:55:51.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.952+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa44075740 con 0x7ffa440ccdd0 2024-03-23T15:55:51.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.973+0000 7ffa39ffb640 1 -- 172.21.15.129:0/3354086985 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ffa3c039090 con 0x7ffa440ccdd0 2024-03-23T15:55:51.976 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:51.976 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:51.976 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:51.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2806b840 msgr2=0x7ffa2806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2806b840 0x7ffa2806dd00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa40000f60 tx=0x7ffa40009040 comp rx=0 tx=0).stop 2024-03-23T15:55:51.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 msgr2=0x7ffa44159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:51.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 0x7ffa44159420 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c02c5e0 tx=0x7ffa3c004a40 comp rx=0 tx=0).stop 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 shutdown_connections 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa440ccdd0 0x7ffa44159420 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ffa2806b840 0x7ffa2806dd00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa440cc9f0 0x7ffa44158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 --2- 172.21.15.129:0/3354086985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa440c8c80 0x7ffa44148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 >> 172.21.15.129:0/3354086985 conn(0x7ffa4406eb20 msgr2=0x7ffa440c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 shutdown_connections 2024-03-23T15:55:51.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:51.975+0000 7ffa4b847640 1 -- 172.21.15.129:0/3354086985 wait complete. 2024-03-23T15:55:52.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:52.593 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:53.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.694+0000 7f10cd8c3640 1 Processor -- start 2024-03-23T15:55:53.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.694+0000 7f10cd8c3640 1 -- start start 2024-03-23T15:55:53.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.695+0000 7f10cd8c3640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f10c81544b0 0x7f10c815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:53.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.695+0000 7f10cd8c3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f10c80c6000 0x7f10c80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:53.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.695+0000 7f10cd8c3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10c80cbcb0 con 0x7f10c80caa00 2024-03-23T15:55:53.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.695+0000 7f10cd8c3640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f10c80c5be0 con 0x7f10c81544b0 2024-03-23T15:55:53.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.695+0000 7f10cd8c3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f10c80c69b0 con 0x7f10c80c6000 2024-03-23T15:55:53.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.696+0000 7f10c77fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f10c80c6000 0x7f10c80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:53.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.696+0000 7f10c6ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f10c81544b0 0x7f10c815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:53.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.696+0000 7f10bffff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f10c80caa00 0x7f10c806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48858/0 (socket says 172.21.15.129:48858) 2024-03-23T15:55:53.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.696+0000 7f10bffff640 1 -- 172.21.15.129:0/354434489 learned_addr learned my addr 172.21.15.129:0/354434489 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.698+0000 7f10c77fe640 1 -- 172.21.15.129:0/354434489 >> v2:172.21.15.129:3300/0 conn(0x7f10c81544b0 msgr2=0x7f10c815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.698+0000 7f10c77fe640 1 --2- 172.21.15.129:0/354434489 >> v2:172.21.15.129:3300/0 conn(0x7f10c81544b0 0x7f10c815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.698+0000 7f10c77fe640 1 -- 172.21.15.129:0/354434489 >> v1:172.21.15.129:6789/0 conn(0x7f10c80caa00 legacy=0x7f10c806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.698+0000 7f10c77fe640 1 -- 172.21.15.129:0/354434489 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c815b280 con 0x7f10c80c6000 2024-03-23T15:55:53.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.700+0000 7f10c77fe640 1 --2- 172.21.15.129:0/354434489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80c6000 0x7f10c80c6470 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10c80b3b50 tx=0x7f10c002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0d68f70496f7f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:53.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.703+0000 7f10c67fc640 1 -- 172.21.15.129:0/354434489 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c002cd00 con 0x7f10c80c6000 2024-03-23T15:55:53.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.705+0000 7f10c67fc640 1 -- 172.21.15.129:0/354434489 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c002ce60 con 0x7f10c80c6000 2024-03-23T15:55:53.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.705+0000 7f10c67fc640 1 -- 172.21.15.129:0/354434489 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0034660 con 0x7f10c80c6000 2024-03-23T15:55:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 -- 172.21.15.129:0/354434489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80c6000 msgr2=0x7f10c80c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/354434489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80c6000 0x7f10c80c6470 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10c80b3b50 tx=0x7f10c002bf50 comp rx=0 tx=0).stop 2024-03-23T15:55:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 -- 172.21.15.129:0/354434489 shutdown_connections 2024-03-23T15:55:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/354434489 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80c6000 0x7f10c80c6470 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/354434489 >> v2:172.21.15.129:3300/0 conn(0x7f10c81544b0 0x7f10c815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 -- 172.21.15.129:0/354434489 >> 172.21.15.129:0/354434489 conn(0x7f10c8061b50 msgr2=0x7f10c8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:53.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 -- 172.21.15.129:0/354434489 shutdown_connections 2024-03-23T15:55:53.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.706+0000 7f10cd8c3640 1 -- 172.21.15.129:0/354434489 wait complete. 2024-03-23T15:55:53.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.708+0000 7f10cd8c3640 1 Processor -- start 2024-03-23T15:55:53.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.708+0000 7f10cd8c3640 1 -- start start 2024-03-23T15:55:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.709+0000 7f10cd8c3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 0x7f10c8146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.709+0000 7f10cd8c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80caa00 0x7f10c8151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.709+0000 7f10cd8c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f10c81544b0 0x7f10c8152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.709+0000 7f10cd8c3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10c81553d0 con 0x7f10c80c6000 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.709+0000 7f10cd8c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f10c80cfd30 con 0x7f10c80caa00 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.709+0000 7f10cd8c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f10c80ce410 con 0x7f10c81544b0 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.710+0000 7f10c6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 0x7f10c8146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.710+0000 7f10bffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80caa00 0x7f10c8151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.710+0000 7f10c77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f10c81544b0 0x7f10c8152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.710+0000 7f10c6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 0x7f10c8146c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47110/0 (socket says 172.21.15.129:47110) 2024-03-23T15:55:53.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.710+0000 7f10c6ffd640 1 -- 172.21.15.129:0/754884432 learned_addr learned my addr 172.21.15.129:0/754884432 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:53.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.713+0000 7f10c6ffd640 1 -- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f10c81544b0 msgr2=0x7f10c8152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:53.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.713+0000 7f10c6ffd640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f10c81544b0 0x7f10c8152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:53.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.713+0000 7f10c6ffd640 1 -- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80caa00 msgr2=0x7f10c8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:53.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.713+0000 7f10c6ffd640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80caa00 0x7f10c8151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:53.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.713+0000 7f10c6ffd640 1 -- 172.21.15.129:0/754884432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c0009a70 con 0x7f10c80c6000 2024-03-23T15:55:53.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.716+0000 7f10c6ffd640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 0x7f10c8146c30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f10b800e870 tx=0x7f10b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.719+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b80094f0 con 0x7f10c80c6000 2024-03-23T15:55:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.719+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b800c940 con 0x7f10c80c6000 2024-03-23T15:55:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.719+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b800ccf0 con 0x7f10c80c6000 2024-03-23T15:55:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.719+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c80cdb70 con 0x7f10c80c6000 2024-03-23T15:55:53.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.719+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c80ce130 con 0x7f10c80c6000 2024-03-23T15:55:53.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.720+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c8075740 con 0x7f10c80c6000 2024-03-23T15:55:53.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.740+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f10b8013470 con 0x7f10c80c6000 2024-03-23T15:55:53.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.740+0000 7f10c4ff9640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f10a806b860 0x7f10a806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.740+0000 7f10bffff640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f10a806b860 0x7f10a806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.741+0000 7f10bffff640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f10a806b860 0x7f10a806dd20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f10c81f3fe0 tx=0x7f10b0007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.997+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f10b8014e40 con 0x7f10c80c6000 2024-03-23T15:55:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:53.997+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f10b8055520 con 0x7f10c80c6000 2024-03-23T15:55:54.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.259+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10c80cb540 con 0x7f10c80c6000 2024-03-23T15:55:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.274+0000 7f10c4ff9640 1 -- 172.21.15.129:0/754884432 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f10b8021090 con 0x7f10c80c6000 2024-03-23T15:55:54.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:54.277 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f10a806b860 msgr2=0x7f10a806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f10a806b860 0x7f10a806dd20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f10c81f3fe0 tx=0x7f10b0007e80 comp rx=0 tx=0).stop 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 msgr2=0x7f10c8146c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 0x7f10c8146c30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f10b800e870 tx=0x7f10b800ed30 comp rx=0 tx=0).stop 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 shutdown_connections 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f10c81544b0 0x7f10c8152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f10a806b860 0x7f10a806dd20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10c80caa00 0x7f10c8151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 --2- 172.21.15.129:0/754884432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c80c6000 0x7f10c8146c30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 >> 172.21.15.129:0/754884432 conn(0x7f10c8061b50 msgr2=0x7f10c806bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:54.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 shutdown_connections 2024-03-23T15:55:54.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:54.276+0000 7f10cd8c3640 1 -- 172.21.15.129:0/754884432 wait complete. 2024-03-23T15:55:54.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:54.898 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:55.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.006+0000 7fa6b48aa640 1 Processor -- start 2024-03-23T15:55:56.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.006+0000 7fa6b48aa640 1 -- start start 2024-03-23T15:55:56.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.007+0000 7fa6b48aa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b00c7e70 0x7fa6b01621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:56.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.007+0000 7fa6b48aa640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6b0073560 con 0x7fa6b0150d00 2024-03-23T15:55:56.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.007+0000 7fa6b48aa640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6b0072fc0 con 0x7fa6b0072ad0 2024-03-23T15:55:56.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.007+0000 7fa6b48aa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b01510e0 con 0x7fa6b00c7e70 2024-03-23T15:55:56.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.008+0000 7fa6ae59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa6b0150d00 0x7fa6b01568b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48862/0 (socket says 172.21.15.129:48862) 2024-03-23T15:55:56.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.008+0000 7fa6ae59c640 1 -- 172.21.15.129:0/2498169331 learned_addr learned my addr 172.21.15.129:0/2498169331 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:55:56.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.008+0000 7fa6aed9d640 1 --2- 172.21.15.129:0/2498169331 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b00c7e70 0x7fa6b01621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:56.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.011+0000 7fa6ad59a640 1 -- 172.21.15.129:0/2498169331 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507913072 0 0) 0x7fa6b0073560 con 0x7fa6b0150d00 2024-03-23T15:55:56.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.011+0000 7fa6ad59a640 1 -- 172.21.15.129:0/2498169331 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa698003520 con 0x7fa6b0150d00 2024-03-23T15:55:56.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.011+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2498169331 >> v1:172.21.15.157:6789/0 conn(0x7fa6b0072ad0 legacy=0x7fa6b006d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.011+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2498169331 >> v1:172.21.15.129:6789/0 conn(0x7fa6b0150d00 legacy=0x7fa6b01568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.011+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2498169331 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b0073280 con 0x7fa6b00c7e70 2024-03-23T15:55:56.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.014+0000 7fa6aed9d640 1 --2- 172.21.15.129:0/2498169331 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b00c7e70 0x7fa6b01621a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009dc0 tx=0x7fa6a802d030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d392f2915a9dfc12 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:56.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.019+0000 7fa6ad59a640 1 -- 172.21.15.129:0/2498169331 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a802dde0 con 0x7fa6b00c7e70 2024-03-23T15:55:56.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.019+0000 7fa6ad59a640 1 -- 172.21.15.129:0/2498169331 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a8035440 con 0x7fa6b00c7e70 2024-03-23T15:55:56.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.019+0000 7fa6ad59a640 1 -- 172.21.15.129:0/2498169331 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a80357d0 con 0x7fa6b00c7e70 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.020+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2498169331 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b00c7e70 msgr2=0x7fa6b01621a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.020+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2498169331 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b00c7e70 0x7fa6b01621a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009dc0 tx=0x7fa6a802d030 comp rx=0 tx=0).stop 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.020+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2498169331 shutdown_connections 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.020+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2498169331 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b00c7e70 0x7fa6b01621a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.020+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2498169331 >> 172.21.15.129:0/2498169331 conn(0x7fa6b00c8370 msgr2=0x7fa6b00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.021+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2498169331 shutdown_connections 2024-03-23T15:55:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.021+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2498169331 wait complete. 2024-03-23T15:55:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.022+0000 7fa6b48aa640 1 Processor -- start 2024-03-23T15:55:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.022+0000 7fa6b48aa640 1 -- start start 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6b48aa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b0072ad0 0x7fa6b0155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6b48aa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b00c7e70 0x7fa6b0149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6b48aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 0x7fa6b0149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6b48aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b014be10 con 0x7fa6b0150d00 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6b48aa640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b014cd30 con 0x7fa6b00c7e70 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6b48aa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b014c220 con 0x7fa6b0072ad0 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6aed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 0x7fa6b0149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6ae59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b0072ad0 0x7fa6b0155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6add9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b00c7e70 0x7fa6b0149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6aed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 0x7fa6b0149b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47120/0 (socket says 172.21.15.129:47120) 2024-03-23T15:55:56.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.023+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2010790023 learned_addr learned my addr 172.21.15.129:0/2010790023 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:56.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.027+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b0072ad0 msgr2=0x7fa6b0155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.027+0000 7fa6aed9d640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b0072ad0 0x7fa6b0155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.027+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b00c7e70 msgr2=0x7fa6b0149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.027+0000 7fa6aed9d640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b00c7e70 0x7fa6b0149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.027+0000 7fa6aed9d640 1 -- 172.21.15.129:0/2010790023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a8009a70 con 0x7fa6b0150d00 2024-03-23T15:55:56.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.029+0000 7fa6aed9d640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 0x7fa6b0149b60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa6a802d640 tx=0x7fa6a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:56.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.033+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8040420 con 0x7fa6b0150d00 2024-03-23T15:55:56.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.033+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a8040580 con 0x7fa6b0150d00 2024-03-23T15:55:56.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.033+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b0148250 con 0x7fa6b0150d00 2024-03-23T15:55:56.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.033+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8040910 con 0x7fa6b0150d00 2024-03-23T15:55:56.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.033+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b0148550 con 0x7fa6b0150d00 2024-03-23T15:55:56.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.034+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b0072710 con 0x7fa6b0150d00 2024-03-23T15:55:56.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.052+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa6a803c040 con 0x7fa6b0150d00 2024-03-23T15:55:56.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.052+0000 7fa69effd640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa69006b810 0x7fa69006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:56.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.053+0000 7fa6ae59c640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa69006b810 0x7fa69006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:56.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.053+0000 7fa6ae59c640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa69006b810 0x7fa69006dcd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0005880 tx=0x7fa6a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:56.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.056+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa6a807b600 con 0x7fa6b0150d00 2024-03-23T15:55:56.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.300+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa6a80778a0 con 0x7fa6b0150d00 2024-03-23T15:55:56.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.523+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6b0075740 con 0x7fa6b0150d00 2024-03-23T15:55:56.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.547+0000 7fa69effd640 1 -- 172.21.15.129:0/2010790023 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa6a804e090 con 0x7fa6b0150d00 2024-03-23T15:55:56.551 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:56.551 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:56.553 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:56.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa69006b810 msgr2=0x7fa69006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa69006b810 0x7fa69006dcd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0005880 tx=0x7fa6a0009040 comp rx=0 tx=0).stop 2024-03-23T15:55:56.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 msgr2=0x7fa6b0149b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:56.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 0x7fa6b0149b60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa6a802d640 tx=0x7fa6a8004a40 comp rx=0 tx=0).stop 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 shutdown_connections 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6b0150d00 0x7fa6b0149b60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa69006b810 0x7fa69006dcd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b00c7e70 0x7fa6b0149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 --2- 172.21.15.129:0/2010790023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b0072ad0 0x7fa6b0155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 >> 172.21.15.129:0/2010790023 conn(0x7fa6b00c8370 msgr2=0x7fa6b014a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:56.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 shutdown_connections 2024-03-23T15:55:56.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:56.549+0000 7fa6b48aa640 1 -- 172.21.15.129:0/2010790023 wait complete. 2024-03-23T15:55:57.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:57.169 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:55:58.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:55:58.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.271+0000 7f8ef63e2640 1 Processor -- start 2024-03-23T15:55:58.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.271+0000 7f8ef63e2640 1 -- start start 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.272+0000 7f8ef63e2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8ef00ccd60 0x7f8ef015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.272+0000 7f8ef63e2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ef00ceb00 0x7f8ef006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.272+0000 7f8ef63e2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef015b290 con 0x7f8ef00cef40 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.272+0000 7f8ef63e2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef00ce800 con 0x7f8ef00ccd60 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.272+0000 7f8ef63e2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef0154a70 con 0x7f8ef00ceb00 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.272+0000 7f8eef7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ef00ceb00 0x7f8ef006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.273+0000 7f8eef7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ef00ceb00 0x7f8ef006c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41636/0 (socket says 172.21.15.129:41636) 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.273+0000 7f8eef7fe640 1 -- 172.21.15.129:0/308388048 learned_addr learned my addr 172.21.15.129:0/308388048 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.273+0000 7f8eeffff640 1 --2- 172.21.15.129:0/308388048 >> v2:172.21.15.129:3300/0 conn(0x7f8ef00ccd60 0x7f8ef015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:58.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.275+0000 7f8eef7fe640 1 -- 172.21.15.129:0/308388048 >> v2:172.21.15.129:3300/0 conn(0x7f8ef00ccd60 msgr2=0x7f8ef015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:55:58.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.275+0000 7f8eef7fe640 1 --2- 172.21.15.129:0/308388048 >> v2:172.21.15.129:3300/0 conn(0x7f8ef00ccd60 0x7f8ef015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.275+0000 7f8eef7fe640 1 -- 172.21.15.129:0/308388048 >> v1:172.21.15.129:6789/0 conn(0x7f8ef00cef40 legacy=0x7f8ef015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:58.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.275+0000 7f8eef7fe640 1 -- 172.21.15.129:0/308388048 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef00cd950 con 0x7f8ef00ceb00 2024-03-23T15:55:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.277+0000 7f8eef7fe640 1 --2- 172.21.15.129:0/308388048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ceb00 0x7f8ef006c270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8ef008d1a0 tx=0x7f8ee402ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75472b9d51e5f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:58.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.282+0000 7f8eeeffd640 1 -- 172.21.15.129:0/308388048 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee402f850 con 0x7f8ef00ceb00 2024-03-23T15:55:58.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.282+0000 7f8eeeffd640 1 -- 172.21.15.129:0/308388048 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee402f9b0 con 0x7f8ef00ceb00 2024-03-23T15:55:58.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.282+0000 7f8eeeffd640 1 -- 172.21.15.129:0/308388048 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee402fd40 con 0x7f8ef00ceb00 2024-03-23T15:55:58.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 -- 172.21.15.129:0/308388048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ceb00 msgr2=0x7f8ef006c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/308388048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ceb00 0x7f8ef006c270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8ef008d1a0 tx=0x7f8ee402ba30 comp rx=0 tx=0).stop 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 -- 172.21.15.129:0/308388048 shutdown_connections 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/308388048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ceb00 0x7f8ef006c270 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/308388048 >> v2:172.21.15.129:3300/0 conn(0x7f8ef00ccd60 0x7f8ef015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 -- 172.21.15.129:0/308388048 >> 172.21.15.129:0/308388048 conn(0x7f8ef00b1840 msgr2=0x7f8ef00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 -- 172.21.15.129:0/308388048 shutdown_connections 2024-03-23T15:55:58.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.283+0000 7f8ef63e2640 1 -- 172.21.15.129:0/308388048 wait complete. 2024-03-23T15:55:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.284+0000 7f8ef63e2640 1 Processor -- start 2024-03-23T15:55:58.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.284+0000 7f8ef63e2640 1 -- start start 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.285+0000 7f8ef63e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 0x7f8ef015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.285+0000 7f8ef63e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef00ceb00 0x7f8ef00cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.285+0000 7f8ef63e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ef00cef40 0x7f8ef01f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.285+0000 7f8ef63e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef00c70b0 con 0x7f8ef00ceb00 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.285+0000 7f8ef63e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01f1be0 con 0x7f8ef00ccd60 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.285+0000 7f8ef63e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef01f15d0 con 0x7f8ef00cef40 2024-03-23T15:55:58.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.286+0000 7f8eeffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 0x7f8ef015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:58.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.286+0000 7f8ef497f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ef00cef40 0x7f8ef01f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:58.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.286+0000 7f8eeffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 0x7f8ef015c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41642/0 (socket says 172.21.15.129:41642) 2024-03-23T15:55:58.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.286+0000 7f8eeffff640 1 -- 172.21.15.129:0/1175249476 learned_addr learned my addr 172.21.15.129:0/1175249476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:55:58.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.288+0000 7f8eef7fe640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef00ceb00 0x7f8ef00cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:58.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.289+0000 7f8eeffff640 1 -- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ef00cef40 msgr2=0x7f8ef01f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:58.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.289+0000 7f8eeffff640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ef00cef40 0x7f8ef01f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.289+0000 7f8eeffff640 1 -- 172.21.15.129:0/1175249476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef00ceb00 msgr2=0x7f8ef00cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:58.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.289+0000 7f8eeffff640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef00ceb00 0x7f8ef00cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.290+0000 7f8eeffff640 1 -- 172.21.15.129:0/1175249476 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee40094d0 con 0x7f8ef00ccd60 2024-03-23T15:55:58.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.292+0000 7f8eeffff640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 0x7f8ef015c200 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000e870 tx=0x7f8ee000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:58.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.296+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee00094f0 con 0x7f8ef00ccd60 2024-03-23T15:55:58.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.297+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee000c940 con 0x7f8ef00ccd60 2024-03-23T15:55:58.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.297+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee000ccf0 con 0x7f8ef00ccd60 2024-03-23T15:55:58.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.297+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef015cfe0 con 0x7f8ef00ccd60 2024-03-23T15:55:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.297+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef015cd00 con 0x7f8ef00ccd60 2024-03-23T15:55:58.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.298+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef004f530 con 0x7f8ef00ccd60 2024-03-23T15:55:58.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.557+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8ee0013470 con 0x7f8ef00ccd60 2024-03-23T15:55:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.557+0000 7f8eecff9640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8edc06b810 0x7f8edc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:55:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.557+0000 7f8eef7fe640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8edc06b810 0x7f8edc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:55:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.558+0000 7f8eef7fe640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8edc06b810 0x7f8edc06dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4036000 tx=0x7f8ee4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:55:58.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.560+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8ee0014e40 con 0x7f8ef00ccd60 2024-03-23T15:55:58.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.575+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8ee0055520 con 0x7f8ef00ccd60 2024-03-23T15:55:58.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.834+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ef006db50 con 0x7f8ef00ccd60 2024-03-23T15:55:58.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.954+0000 7f8eecff9640 1 -- 172.21.15.129:0/1175249476 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8ee0021090 con 0x7f8ef00ccd60 2024-03-23T15:55:58.956 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:55:58.957 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:55:58.957 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8edc06b810 msgr2=0x7f8edc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8edc06b810 0x7f8edc06dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4036000 tx=0x7f8ee4008030 comp rx=0 tx=0).stop 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 msgr2=0x7f8ef015c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 0x7f8ef015c200 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000e870 tx=0x7f8ee000ed30 comp rx=0 tx=0).stop 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 shutdown_connections 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ef00cef40 0x7f8ef01f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef00ceb00 0x7f8ef00cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8edc06b810 0x7f8edc06dcd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 --2- 172.21.15.129:0/1175249476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ef00ccd60 0x7f8ef015c200 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 >> 172.21.15.129:0/1175249476 conn(0x7f8ef00b1840 msgr2=0x7f8ef01e5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:55:58.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 shutdown_connections 2024-03-23T15:55:58.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:55:58.957+0000 7f8ef63e2640 1 -- 172.21.15.129:0/1175249476 wait complete. 2024-03-23T15:55:59.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:55:59.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:00.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:00.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.679+0000 7fcc3b655640 1 Processor -- start 2024-03-23T15:56:00.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.679+0000 7fcc3b655640 1 -- start start 2024-03-23T15:56:00.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.680+0000 7fcc3b655640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc340ccd60 0x7fcc3415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.680+0000 7fcc3b655640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3415b290 con 0x7fcc340cef40 2024-03-23T15:56:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.680+0000 7fcc3b655640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340ce800 con 0x7fcc340ceb00 2024-03-23T15:56:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.680+0000 7fcc3b655640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fcc34154a70 con 0x7fcc340ccd60 2024-03-23T15:56:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.681+0000 7fcc38bf0640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fcc340ceb00 0x7fcc3406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:47366/0 (socket says 172.21.15.129:47366) 2024-03-23T15:56:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.681+0000 7fcc38bf0640 1 -- 172.21.15.129:0/3716690387 learned_addr learned my addr 172.21.15.129:0/3716690387 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.681+0000 7fcc393f1640 1 --2- 172.21.15.129:0/3716690387 >> v2:172.21.15.129:3300/0 conn(0x7fcc340ccd60 0x7fcc3415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:00.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.684+0000 7fcc2bfff640 1 -- 172.21.15.129:0/3716690387 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453318088 0 0) 0x7fcc340ce800 con 0x7fcc340ceb00 2024-03-23T15:56:00.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.684+0000 7fcc2bfff640 1 -- 172.21.15.129:0/3716690387 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc20003520 con 0x7fcc340ceb00 2024-03-23T15:56:00.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.684+0000 7fcc393f1640 1 -- 172.21.15.129:0/3716690387 >> v1:172.21.15.157:6789/0 conn(0x7fcc340ceb00 legacy=0x7fcc3406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:00.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.684+0000 7fcc393f1640 1 -- 172.21.15.129:0/3716690387 >> v1:172.21.15.129:6789/0 conn(0x7fcc340cef40 legacy=0x7fcc3415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:00.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.684+0000 7fcc393f1640 1 -- 172.21.15.129:0/3716690387 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340cd950 con 0x7fcc340ccd60 2024-03-23T15:56:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.689+0000 7fcc393f1640 1 --2- 172.21.15.129:0/3716690387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc3415ad50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcc300097a0 tx=0x7fcc3002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd936f77debefe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:00.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.692+0000 7fcc2bfff640 1 -- 172.21.15.129:0/3716690387 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002f810 con 0x7fcc340ccd60 2024-03-23T15:56:00.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.692+0000 7fcc2bfff640 1 -- 172.21.15.129:0/3716690387 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc3002f970 con 0x7fcc340ccd60 2024-03-23T15:56:00.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.692+0000 7fcc2bfff640 1 -- 172.21.15.129:0/3716690387 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002fd00 con 0x7fcc340ccd60 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 -- 172.21.15.129:0/3716690387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 msgr2=0x7fcc3415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 --2- 172.21.15.129:0/3716690387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc3415ad50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcc300097a0 tx=0x7fcc3002b9f0 comp rx=0 tx=0).stop 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 -- 172.21.15.129:0/3716690387 shutdown_connections 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 --2- 172.21.15.129:0/3716690387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc3415ad50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 -- 172.21.15.129:0/3716690387 >> 172.21.15.129:0/3716690387 conn(0x7fcc340b1840 msgr2=0x7fcc340b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 -- 172.21.15.129:0/3716690387 shutdown_connections 2024-03-23T15:56:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.693+0000 7fcc3b655640 1 -- 172.21.15.129:0/3716690387 wait complete. 2024-03-23T15:56:00.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 Processor -- start 2024-03-23T15:56:00.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 -- start start 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc341f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc340ceb00 0x7fcc341575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc340cef40 0x7fcc340c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34147810 con 0x7fcc340ccd60 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34159950 con 0x7fcc340ceb00 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.695+0000 7fcc3b655640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc34154de0 con 0x7fcc340cef40 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.696+0000 7fcc393f1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc341f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.696+0000 7fcc38bf0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc340ceb00 0x7fcc341575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.696+0000 7fcc393f1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc341f1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57610/0 (socket says 172.21.15.129:57610) 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.696+0000 7fcc393f1640 1 -- 172.21.15.129:0/2745842558 learned_addr learned my addr 172.21.15.129:0/2745842558 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.696+0000 7fcc39bf2640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc340cef40 0x7fcc340c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.699+0000 7fcc393f1640 1 -- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc340cef40 msgr2=0x7fcc340c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.699+0000 7fcc393f1640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc340cef40 0x7fcc340c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.699+0000 7fcc393f1640 1 -- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc340ceb00 msgr2=0x7fcc341575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.699+0000 7fcc393f1640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc340ceb00 0x7fcc341575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.699+0000 7fcc393f1640 1 -- 172.21.15.129:0/2745842558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc30009400 con 0x7fcc340ccd60 2024-03-23T15:56:00.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.701+0000 7fcc393f1640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc341f1110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcc3002c000 tx=0x7fcc30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:00.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.705+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3003c040 con 0x7fcc340ccd60 2024-03-23T15:56:00.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.705+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc300378f0 con 0x7fcc340ccd60 2024-03-23T15:56:00.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.705+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30037ca0 con 0x7fcc340ccd60 2024-03-23T15:56:00.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.705+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc341550c0 con 0x7fcc340ccd60 2024-03-23T15:56:00.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.705+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc34155680 con 0x7fcc340ccd60 2024-03-23T15:56:00.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.706+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3404f530 con 0x7fcc340ccd60 2024-03-23T15:56:00.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.961+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc3002d3a0 con 0x7fcc340ccd60 2024-03-23T15:56:00.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.961+0000 7fcc29ffb640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc1806b860 0x7fcc1806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:00.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.962+0000 7fcc38bf0640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc1806b860 0x7fcc1806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:00.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.962+0000 7fcc38bf0640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc1806b860 0x7fcc1806dd20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcc240011e0 tx=0x7fcc24007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.981+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcc3007abd0 con 0x7fcc340ccd60 2024-03-23T15:56:00.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:00.981+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc30076e70 con 0x7fcc340ccd60 2024-03-23T15:56:01.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.254+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc340686f0 con 0x7fcc340ccd60 2024-03-23T15:56:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.275+0000 7fcc29ffb640 1 -- 172.21.15.129:0/2745842558 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcc30047020 con 0x7fcc340ccd60 2024-03-23T15:56:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:01.278 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc1806b860 msgr2=0x7fcc1806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc1806b860 0x7fcc1806dd20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcc240011e0 tx=0x7fcc24007d30 comp rx=0 tx=0).stop 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 msgr2=0x7fcc341f1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc341f1110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcc3002c000 tx=0x7fcc30004730 comp rx=0 tx=0).stop 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 shutdown_connections 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc340cef40 0x7fcc340c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc1806b860 0x7fcc1806dd20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc340ceb00 0x7fcc341575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 --2- 172.21.15.129:0/2745842558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc340ccd60 0x7fcc341f1110 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 >> 172.21.15.129:0/2745842558 conn(0x7fcc340b1840 msgr2=0x7fcc34158530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 shutdown_connections 2024-03-23T15:56:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:01.277+0000 7fcc3b655640 1 -- 172.21.15.129:0/2745842558 wait complete. 2024-03-23T15:56:01.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:01.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:02.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:02.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.995+0000 7f3d40c76640 1 Processor -- start 2024-03-23T15:56:02.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.995+0000 7f3d40c76640 1 -- start start 2024-03-23T15:56:02.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.996+0000 7f3d40c76640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3d3c15c000 0x7f3d3c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:02.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.996+0000 7f3d40c76640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3c15b290 con 0x7f3d3c0ccd60 2024-03-23T15:56:02.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.996+0000 7f3d40c76640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3c0ce800 con 0x7f3d3c0ceb00 2024-03-23T15:56:02.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.996+0000 7f3d40c76640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3c154a70 con 0x7f3d3c15c000 2024-03-23T15:56:02.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.996+0000 7f3d3a59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3d3c0ccd60 0x7f3d3c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:53296/0 (socket says 172.21.15.129:53296) 2024-03-23T15:56:02.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.996+0000 7f3d3a59c640 1 -- 172.21.15.129:0/1760145274 learned_addr learned my addr 172.21.15.129:0/1760145274 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:02.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.997+0000 7f3d3ad9d640 1 --2- 172.21.15.129:0/1760145274 >> v2:172.21.15.157:3300/0 conn(0x7f3d3c15c000 0x7f3d3c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.999+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3895323976 0 0) 0x7f3d2c001110 con 0x7f3d3c0ccd60 2024-03-23T15:56:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:02.999+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d24003520 con 0x7f3d3c0ccd60 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2432831803 0 0) 0x7f3d24003520 con 0x7f3d3c0ccd60 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 >> v2:172.21.15.157:3300/0 conn(0x7f3d3c15c000 msgr2=0x7f3d3c15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 --2- 172.21.15.129:0/1760145274 >> v2:172.21.15.157:3300/0 conn(0x7f3d3c15c000 0x7f3d3c15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 >> v1:172.21.15.157:6789/0 conn(0x7f3d3c0ceb00 legacy=0x7f3d3c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0cd950 con 0x7f3d3c0ccd60 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f3d2c001800 con 0x7f3d3c0ccd60 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 >> v1:172.21.15.129:6789/0 conn(0x7f3d3c0ccd60 legacy=0x7f3d3c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d24003970 0x7f3d24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240047c0 0x7f3d24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24005180 0x7f3d2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c154a70 con 0x7f3d240047c0 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2400c030 con 0x7f3d24005180 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2400c1f0 con 0x7f3d24003970 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.003+0000 7f3d39d9b640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240047c0 0x7f3d24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.004+0000 7f3d3a59c640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d24003970 0x7f3d24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d3a59c640 1 -- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24005180 msgr2=0x7f3d2400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:03.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d3a59c640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24005180 0x7f3d2400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d3a59c640 1 -- 172.21.15.129:0/1760145274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240047c0 msgr2=0x7f3d24004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:03.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d3a59c640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240047c0 0x7f3d24004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d3a59c640 1 -- 172.21.15.129:0/1760145274 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0cd950 con 0x7f3d24003970 2024-03-23T15:56:03.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d3ad9d640 1 -- 172.21.15.129:0/1760145274 reap_dead start 2024-03-23T15:56:03.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.008+0000 7f3d39d9b640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240047c0 0x7f3d24004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:56:03.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.010+0000 7f3d3a59c640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d24003970 0x7f3d24003d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00c1b0 tx=0x7f3d240057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d1547d76beabc56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:03.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.015+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c005ad0 con 0x7f3d24003970 2024-03-23T15:56:03.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.016+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c005c50 con 0x7f3d24003970 2024-03-23T15:56:03.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.016+0000 7f3d3959a640 1 -- 172.21.15.129:0/1760145274 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c02c870 con 0x7f3d24003970 2024-03-23T15:56:03.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.016+0000 7f3d40c76640 1 -- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d24003970 msgr2=0x7f3d24003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.016+0000 7f3d40c76640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d24003970 0x7f3d24003d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00c1b0 tx=0x7f3d240057f0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.017+0000 7f3d40c76640 1 -- 172.21.15.129:0/1760145274 shutdown_connections 2024-03-23T15:56:03.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.017+0000 7f3d40c76640 1 --2- 172.21.15.129:0/1760145274 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d24003970 0x7f3d24003d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.017+0000 7f3d40c76640 1 -- 172.21.15.129:0/1760145274 >> 172.21.15.129:0/1760145274 conn(0x7f3d3c0b1840 msgr2=0x7f3d3c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:03.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.017+0000 7f3d40c76640 1 -- 172.21.15.129:0/1760145274 shutdown_connections 2024-03-23T15:56:03.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.017+0000 7f3d40c76640 1 -- 172.21.15.129:0/1760145274 wait complete. 2024-03-23T15:56:03.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.019+0000 7f3d40c76640 1 Processor -- start 2024-03-23T15:56:03.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.019+0000 7f3d40c76640 1 -- start start 2024-03-23T15:56:03.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.020+0000 7f3d40c76640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24003970 0x7f3d3c0b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.020+0000 7f3d40c76640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d3c151f90 0x7f3d3c15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.020+0000 7f3d40c76640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 0x7f3d3c14a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.020+0000 7f3d40c76640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0c2d20 con 0x7f3d3c149bd0 2024-03-23T15:56:03.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.020+0000 7f3d40c76640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c153250 con 0x7f3d24003970 2024-03-23T15:56:03.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.020+0000 7f3d40c76640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c14a590 con 0x7f3d3c151f90 2024-03-23T15:56:03.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.021+0000 7f3d3ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 0x7f3d3c14a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.021+0000 7f3d39d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d3c151f90 0x7f3d3c15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.021+0000 7f3d3a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24003970 0x7f3d3c0b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.021+0000 7f3d3ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 0x7f3d3c14a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57640/0 (socket says 172.21.15.129:57640) 2024-03-23T15:56:03.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.021+0000 7f3d3ad9d640 1 -- 172.21.15.129:0/772287517 learned_addr learned my addr 172.21.15.129:0/772287517 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:03.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.026+0000 7f3d3ad9d640 1 -- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d3c151f90 msgr2=0x7f3d3c15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.026+0000 7f3d3ad9d640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d3c151f90 0x7f3d3c15cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.026+0000 7f3d3ad9d640 1 -- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24003970 msgr2=0x7f3d3c0b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.026+0000 7f3d3ad9d640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24003970 0x7f3d3c0b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.026+0000 7f3d3ad9d640 1 -- 172.21.15.129:0/772287517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c153c90 con 0x7f3d3c149bd0 2024-03-23T15:56:03.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.029+0000 7f3d3ad9d640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 0x7f3d3c14a050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0c3330 tx=0x7f3d3400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.033+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34010040 con 0x7f3d3c149bd0 2024-03-23T15:56:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.033+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3400bc80 con 0x7f3d3c149bd0 2024-03-23T15:56:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.033+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c152cd0 con 0x7f3d3c149bd0 2024-03-23T15:56:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.033+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34014400 con 0x7f3d3c149bd0 2024-03-23T15:56:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.033+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c0b28f0 con 0x7f3d3c149bd0 2024-03-23T15:56:03.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.034+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c04f530 con 0x7f3d3c149bd0 2024-03-23T15:56:03.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.299+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3d34014560 con 0x7f3d3c149bd0 2024-03-23T15:56:03.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.300+0000 7f3d2affd640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3d1c06b860 0x7f3d1c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:03.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.300+0000 7f3d3a59c640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3d1c06b860 0x7f3d1c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:03.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.301+0000 7f3d3a59c640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3d1c06b860 0x7f3d1c06dd20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c034000 tx=0x7f3d2c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.316+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3d340597c0 con 0x7f3d3c149bd0 2024-03-23T15:56:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.317+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3d34055a60 con 0x7f3d3c149bd0 2024-03-23T15:56:03.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.553+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d3c0686f0 con 0x7f3d3c149bd0 2024-03-23T15:56:03.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.575+0000 7f3d2affd640 1 -- 172.21.15.129:0/772287517 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3d34059120 con 0x7f3d3c149bd0 2024-03-23T15:56:03.578 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:03.578 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:03.578 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:03.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3d1c06b860 msgr2=0x7f3d1c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3d1c06b860 0x7f3d1c06dd20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c034000 tx=0x7f3d2c009030 comp rx=0 tx=0).stop 2024-03-23T15:56:03.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 msgr2=0x7f3d3c14a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 0x7f3d3c14a050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0c3330 tx=0x7f3d3400b480 comp rx=0 tx=0).stop 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 shutdown_connections 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d3c149bd0 0x7f3d3c14a050 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d3c151f90 0x7f3d3c15cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3d1c06b860 0x7f3d1c06dd20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 --2- 172.21.15.129:0/772287517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d24003970 0x7f3d3c0b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.577+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 >> 172.21.15.129:0/772287517 conn(0x7f3d3c0b1840 msgr2=0x7f3d3c14d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.578+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 shutdown_connections 2024-03-23T15:56:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:03.578+0000 7f3d40c76640 1 -- 172.21.15.129:0/772287517 wait complete. 2024-03-23T15:56:04.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:04.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:05.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:05.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.298+0000 7fb36891e640 1 Processor -- start 2024-03-23T15:56:05.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.298+0000 7fb36891e640 1 -- start start 2024-03-23T15:56:05.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.299+0000 7fb36891e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3640ccd60 0x7fb36415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:05.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.299+0000 7fb36891e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb3640cef40 0x7fb36415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:05.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.299+0000 7fb36891e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36415b290 con 0x7fb3640ceb00 2024-03-23T15:56:05.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.299+0000 7fb36891e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb3640ce800 con 0x7fb3640ccd60 2024-03-23T15:56:05.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.299+0000 7fb36891e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb364154a70 con 0x7fb3640cef40 2024-03-23T15:56:05.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.300+0000 7fb36259c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb3640ccd60 0x7fb36415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:05.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.300+0000 7fb361d9b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb3640ceb00 0x7fb36406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:53310/0 (socket says 172.21.15.129:53310) 2024-03-23T15:56:05.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.300+0000 7fb361d9b640 1 -- 172.21.15.129:0/3393765253 learned_addr learned my addr 172.21.15.129:0/3393765253 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:05.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.303+0000 7fb36159a640 1 -- 172.21.15.129:0/3393765253 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400263222 0 0) 0x7fb358001110 con 0x7fb3640ceb00 2024-03-23T15:56:05.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.303+0000 7fb36159a640 1 -- 172.21.15.129:0/3393765253 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb34c003520 con 0x7fb3640ceb00 2024-03-23T15:56:05.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.308+0000 7fb36259c640 1 -- 172.21.15.129:0/3393765253 >> v2:172.21.15.157:3300/0 conn(0x7fb3640cef40 msgr2=0x7fb36415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:05.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.308+0000 7fb36259c640 1 --2- 172.21.15.129:0/3393765253 >> v2:172.21.15.157:3300/0 conn(0x7fb3640cef40 0x7fb36415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.308+0000 7fb36259c640 1 -- 172.21.15.129:0/3393765253 >> v1:172.21.15.129:6789/0 conn(0x7fb3640ceb00 legacy=0x7fb36406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:05.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.308+0000 7fb36259c640 1 -- 172.21.15.129:0/3393765253 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3640cd950 con 0x7fb3640ccd60 2024-03-23T15:56:05.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.312+0000 7fb36259c640 1 --2- 172.21.15.129:0/3393765253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 0x7fb36415ad50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb3640cf660 tx=0x7fb35402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877aae13708ca2a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:05.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.317+0000 7fb36159a640 1 -- 172.21.15.129:0/3393765253 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354030610 con 0x7fb3640ccd60 2024-03-23T15:56:05.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.317+0000 7fb36159a640 1 -- 172.21.15.129:0/3393765253 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb354030770 con 0x7fb3640ccd60 2024-03-23T15:56:05.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.317+0000 7fb36159a640 1 -- 172.21.15.129:0/3393765253 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354030b00 con 0x7fb3640ccd60 2024-03-23T15:56:05.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.317+0000 7fb36891e640 1 -- 172.21.15.129:0/3393765253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 msgr2=0x7fb36415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:05.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.317+0000 7fb36891e640 1 --2- 172.21.15.129:0/3393765253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 0x7fb36415ad50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb3640cf660 tx=0x7fb35402b960 comp rx=0 tx=0).stop 2024-03-23T15:56:05.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.318+0000 7fb36891e640 1 -- 172.21.15.129:0/3393765253 shutdown_connections 2024-03-23T15:56:05.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.318+0000 7fb36891e640 1 --2- 172.21.15.129:0/3393765253 >> v2:172.21.15.157:3300/0 conn(0x7fb3640cef40 0x7fb36415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.318+0000 7fb36891e640 1 --2- 172.21.15.129:0/3393765253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 0x7fb36415ad50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.318+0000 7fb36891e640 1 -- 172.21.15.129:0/3393765253 >> 172.21.15.129:0/3393765253 conn(0x7fb3640b1840 msgr2=0x7fb3640b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:05.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.318+0000 7fb36891e640 1 -- 172.21.15.129:0/3393765253 shutdown_connections 2024-03-23T15:56:05.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.318+0000 7fb36891e640 1 -- 172.21.15.129:0/3393765253 wait complete. 2024-03-23T15:56:05.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.319+0000 7fb36891e640 1 Processor -- start 2024-03-23T15:56:05.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.319+0000 7fb36891e640 1 -- start start 2024-03-23T15:56:05.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.320+0000 7fb36891e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 0x7fb364154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:05.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.320+0000 7fb36891e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3640ceb00 0x7fb3641563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:05.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.320+0000 7fb36891e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3640cef40 0x7fb364159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:05.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.320+0000 7fb36891e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3640cb3e0 con 0x7fb3640ccd60 2024-03-23T15:56:05.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.320+0000 7fb36891e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb364155b90 con 0x7fb3640cef40 2024-03-23T15:56:05.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.320+0000 7fb36891e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb364156900 con 0x7fb3640ceb00 2024-03-23T15:56:05.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.321+0000 7fb361d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3640ceb00 0x7fb3641563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:05.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.321+0000 7fb362d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3640cef40 0x7fb364159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:05.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.322+0000 7fb361d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3640ceb00 0x7fb3641563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:40504/0 (socket says 172.21.15.129:40504) 2024-03-23T15:56:05.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.322+0000 7fb361d9b640 1 -- 172.21.15.129:0/3011912136 learned_addr learned my addr 172.21.15.129:0/3011912136 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:05.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.326+0000 7fb362d9d640 1 -- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3640ceb00 msgr2=0x7fb3641563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:05.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.326+0000 7fb362d9d640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3640ceb00 0x7fb3641563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.326+0000 7fb362d9d640 1 -- 172.21.15.129:0/3011912136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 msgr2=0x7fb364154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:05.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.326+0000 7fb362d9d640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 0x7fb364154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.326+0000 7fb362d9d640 1 -- 172.21.15.129:0/3011912136 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb354009400 con 0x7fb3640cef40 2024-03-23T15:56:05.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.328+0000 7fb362d9d640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3640cef40 0x7fb364159310 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00be50 tx=0x7fb35c00bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:05.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.332+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00ce90 con 0x7fb3640cef40 2024-03-23T15:56:05.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.332+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb35c014440 con 0x7fb3640cef40 2024-03-23T15:56:05.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.332+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb364155610 con 0x7fb3640cef40 2024-03-23T15:56:05.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.332+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0147d0 con 0x7fb3640cef40 2024-03-23T15:56:05.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.332+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3641f0240 con 0x7fb3640cef40 2024-03-23T15:56:05.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.334+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36404f530 con 0x7fb3640cef40 2024-03-23T15:56:05.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.597+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb35c01d3b0 con 0x7fb3640cef40 2024-03-23T15:56:05.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.597+0000 7fb352ffd640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb34406b810 0x7fb34406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:05.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.598+0000 7fb36259c640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb34406b810 0x7fb34406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:05.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.598+0000 7fb36259c640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb34406b810 0x7fb34406dcd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb354036000 tx=0x7fb354030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:05.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.605+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb35c058fd0 con 0x7fb3640cef40 2024-03-23T15:56:05.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.618+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb35c055270 con 0x7fb3640cef40 2024-03-23T15:56:05.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.845+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3640686f0 con 0x7fb3640cef40 2024-03-23T15:56:05.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.858+0000 7fb352ffd640 1 -- 172.21.15.129:0/3011912136 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb35c019030 con 0x7fb3640cef40 2024-03-23T15:56:05.859 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:05.859 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:05.860 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb34406b810 msgr2=0x7fb34406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb34406b810 0x7fb34406dcd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb354036000 tx=0x7fb354030000 comp rx=0 tx=0).stop 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3640cef40 msgr2=0x7fb364159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3640cef40 0x7fb364159310 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00be50 tx=0x7fb35c00bf40 comp rx=0 tx=0).stop 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 shutdown_connections 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb34406b810 0x7fb34406dcd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3640cef40 0x7fb364159310 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3640ceb00 0x7fb3641563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 --2- 172.21.15.129:0/3011912136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3640ccd60 0x7fb364154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:05.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 >> 172.21.15.129:0/3011912136 conn(0x7fb3640b1840 msgr2=0x7fb36415a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:05.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 shutdown_connections 2024-03-23T15:56:05.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:05.860+0000 7fb36891e640 1 -- 172.21.15.129:0/3011912136 wait complete. 2024-03-23T15:56:06.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:06.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:07.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:07.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.581+0000 7ff549e64640 1 Processor -- start 2024-03-23T15:56:07.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.581+0000 7ff549e64640 1 -- start start 2024-03-23T15:56:07.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.581+0000 7ff549e64640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff5440ccd60 0x7ff54415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:07.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.581+0000 7ff549e64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440cef40 0x7ff54415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:07.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff549e64640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54415b290 con 0x7ff5440ceb00 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff549e64640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ff5440ce800 con 0x7ff5440ccd60 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff549e64640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff544154a70 con 0x7ff5440cef40 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff5437fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff5440ccd60 0x7ff54415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff5437fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff5440ccd60 0x7ff54415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57668/0 (socket says 172.21.15.129:57668) 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff5437fe640 1 -- 172.21.15.129:0/82510425 learned_addr learned my addr 172.21.15.129:0/82510425 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff542ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff5440ceb00 0x7ff54406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:47390/0 (socket says 172.21.15.129:47390) 2024-03-23T15:56:07.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.582+0000 7ff543fff640 1 --2- 172.21.15.129:0/82510425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440cef40 0x7ff54415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:07.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.585+0000 7ff5437fe640 1 -- 172.21.15.129:0/82510425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440cef40 msgr2=0x7ff54415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:07.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.585+0000 7ff5437fe640 1 --2- 172.21.15.129:0/82510425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440cef40 0x7ff54415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:07.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.585+0000 7ff5437fe640 1 -- 172.21.15.129:0/82510425 >> v1:172.21.15.157:6789/0 conn(0x7ff5440ceb00 legacy=0x7ff54406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:07.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.585+0000 7ff5437fe640 1 -- 172.21.15.129:0/82510425 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5440cd950 con 0x7ff5440ccd60 2024-03-23T15:56:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.586+0000 7ff5437fe640 1 --2- 172.21.15.129:0/82510425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff54415ad50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff534009870 tx=0x7ff53402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd94785106bd39d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.590+0000 7ff5427fc640 1 -- 172.21.15.129:0/82510425 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53402f840 con 0x7ff5440ccd60 2024-03-23T15:56:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.590+0000 7ff5427fc640 1 -- 172.21.15.129:0/82510425 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53402f9a0 con 0x7ff5440ccd60 2024-03-23T15:56:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.590+0000 7ff5427fc640 1 -- 172.21.15.129:0/82510425 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53402fd30 con 0x7ff5440ccd60 2024-03-23T15:56:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 -- 172.21.15.129:0/82510425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 msgr2=0x7ff54415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 --2- 172.21.15.129:0/82510425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff54415ad50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff534009870 tx=0x7ff53402ba20 comp rx=0 tx=0).stop 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 -- 172.21.15.129:0/82510425 shutdown_connections 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 --2- 172.21.15.129:0/82510425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440cef40 0x7ff54415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 --2- 172.21.15.129:0/82510425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff54415ad50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 -- 172.21.15.129:0/82510425 >> 172.21.15.129:0/82510425 conn(0x7ff5440b1840 msgr2=0x7ff5440b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 -- 172.21.15.129:0/82510425 shutdown_connections 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.591+0000 7ff549e64640 1 -- 172.21.15.129:0/82510425 wait complete. 2024-03-23T15:56:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.592+0000 7ff549e64640 1 Processor -- start 2024-03-23T15:56:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 -- start start 2024-03-23T15:56:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff5441e3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440ceb00 0x7ff5441e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5440cef40 0x7ff5441f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5441e3640 con 0x7ff5440ccd60 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff54415a010 con 0x7ff5440cef40 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.593+0000 7ff549e64640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff54415a2d0 con 0x7ff5440ceb00 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.594+0000 7ff543fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5440cef40 0x7ff5441f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.594+0000 7ff543fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5440cef40 0x7ff5441f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:50728/0 (socket says 172.21.15.129:50728) 2024-03-23T15:56:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.594+0000 7ff543fff640 1 -- 172.21.15.129:0/3721465020 learned_addr learned my addr 172.21.15.129:0/3721465020 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.595+0000 7ff5437fe640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff5441e3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.597+0000 7ff542ffd640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440ceb00 0x7ff5441e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.598+0000 7ff5437fe640 1 -- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440ceb00 msgr2=0x7ff5441e42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.598+0000 7ff5437fe640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440ceb00 0x7ff5441e42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.598+0000 7ff5437fe640 1 -- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5440cef40 msgr2=0x7ff5441f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.598+0000 7ff5437fe640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5440cef40 0x7ff5441f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.598+0000 7ff5437fe640 1 -- 172.21.15.129:0/3721465020 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5340094d0 con 0x7ff5440ccd60 2024-03-23T15:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.601+0000 7ff5437fe640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff5441e3db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff534033000 tx=0x7ff534004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.605+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534037bc0 con 0x7ff5440ccd60 2024-03-23T15:56:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.605+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff534037d20 con 0x7ff5440ccd60 2024-03-23T15:56:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.605+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54415a5b0 con 0x7ff5440ccd60 2024-03-23T15:56:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.605+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534041640 con 0x7ff5440ccd60 2024-03-23T15:56:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.605+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5441e3890 con 0x7ff5440ccd60 2024-03-23T15:56:07.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.606+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54404f530 con 0x7ff5440ccd60 2024-03-23T15:56:07.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.628+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff5340417a0 con 0x7ff5440ccd60 2024-03-23T15:56:07.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.628+0000 7ff52bfff640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff53006b860 0x7ff53006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:07.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.629+0000 7ff542ffd640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff53006b860 0x7ff53006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:07.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.629+0000 7ff542ffd640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff53006b860 0x7ff53006dd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5440c97c0 tx=0x7ff538007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.898+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff53403c070 con 0x7ff5440ccd60 2024-03-23T15:56:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:07.898+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff5340785a0 con 0x7ff5440ccd60 2024-03-23T15:56:08.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.116+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff54406db50 con 0x7ff5440ccd60 2024-03-23T15:56:08.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.130+0000 7ff52bfff640 1 -- 172.21.15.129:0/3721465020 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff53407bc60 con 0x7ff5440ccd60 2024-03-23T15:56:08.133 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:08.133 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:08.134 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff53006b860 msgr2=0x7ff53006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff53006b860 0x7ff53006dd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5440c97c0 tx=0x7ff538007e80 comp rx=0 tx=0).stop 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 msgr2=0x7ff5441e3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff5441e3db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff534033000 tx=0x7ff534004730 comp rx=0 tx=0).stop 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 shutdown_connections 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff53006b860 0x7ff53006dd20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5440cef40 0x7ff5441f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5440ceb00 0x7ff5441e42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.132+0000 7ff549e64640 1 --2- 172.21.15.129:0/3721465020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5440ccd60 0x7ff5441e3db0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.133+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 >> 172.21.15.129:0/3721465020 conn(0x7ff5440b1840 msgr2=0x7ff544153f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.133+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 shutdown_connections 2024-03-23T15:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:08.133+0000 7ff549e64640 1 -- 172.21.15.129:0/3721465020 wait complete. 2024-03-23T15:56:08.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:56:08.242 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:56:08.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:08.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:09.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:09.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 Processor -- start 2024-03-23T15:56:09.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 -- start start 2024-03-23T15:56:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e80ca4a0 0x7f24e806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f24e815dda0 0x7f24e815e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e80cb810 con 0x7f24e815d9c0 2024-03-23T15:56:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f24e815e770 con 0x7f24e815dda0 2024-03-23T15:56:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.852+0000 7f24efd0c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24e80cc3b0 con 0x7f24e80ca4a0 2024-03-23T15:56:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.853+0000 7f24edaa8640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f24e815d9c0 0x7f24e80cbca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:53314/0 (socket says 172.21.15.129:53314) 2024-03-23T15:56:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.853+0000 7f24edaa8640 1 -- 172.21.15.129:0/1277649366 learned_addr learned my addr 172.21.15.129:0/1277649366 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.853+0000 7f24ee2a9640 1 --2- 172.21.15.129:0/1277649366 >> v2:172.21.15.157:3300/0 conn(0x7f24e815dda0 0x7f24e815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.854+0000 7f24ed2a7640 1 --2- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e80ca4a0 0x7f24e806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:09.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.855+0000 7f24ecaa6640 1 -- 172.21.15.129:0/1277649366 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238082336 0 0) 0x7f24e80cb810 con 0x7f24e815d9c0 2024-03-23T15:56:09.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.855+0000 7f24ecaa6640 1 -- 172.21.15.129:0/1277649366 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24d4003520 con 0x7f24e815d9c0 2024-03-23T15:56:09.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.857+0000 7f24ee2a9640 1 -- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e80ca4a0 msgr2=0x7f24e806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:09.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.857+0000 7f24ee2a9640 1 --2- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e80ca4a0 0x7f24e806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:09.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.857+0000 7f24ee2a9640 1 -- 172.21.15.129:0/1277649366 >> v1:172.21.15.129:6789/0 conn(0x7f24e815d9c0 legacy=0x7f24e80cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:09.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.857+0000 7f24ee2a9640 1 -- 172.21.15.129:0/1277649366 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e80cac00 con 0x7f24e815dda0 2024-03-23T15:56:09.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.860+0000 7f24ee2a9640 1 --2- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e815e230 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009d90 tx=0x7f24e002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b5f7305019a894 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:09.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.865+0000 7f24ecaa6640 1 -- 172.21.15.129:0/1277649366 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e002cc50 con 0x7f24e815dda0 2024-03-23T15:56:09.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.865+0000 7f24ecaa6640 1 -- 172.21.15.129:0/1277649366 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e002cdb0 con 0x7f24e815dda0 2024-03-23T15:56:09.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.865+0000 7f24ecaa6640 1 -- 172.21.15.129:0/1277649366 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0034660 con 0x7f24e815dda0 2024-03-23T15:56:09.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 -- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 msgr2=0x7f24e815e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 --2- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e815e230 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009d90 tx=0x7f24e002bea0 comp rx=0 tx=0).stop 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 -- 172.21.15.129:0/1277649366 shutdown_connections 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 --2- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e815e230 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 --2- 172.21.15.129:0/1277649366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e80ca4a0 0x7f24e806c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 -- 172.21.15.129:0/1277649366 >> 172.21.15.129:0/1277649366 conn(0x7f24e80cfb20 msgr2=0x7f24e80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 -- 172.21.15.129:0/1277649366 shutdown_connections 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.866+0000 7f24efd0c640 1 -- 172.21.15.129:0/1277649366 wait complete. 2024-03-23T15:56:09.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.867+0000 7f24efd0c640 1 Processor -- start 2024-03-23T15:56:09.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.867+0000 7f24efd0c640 1 -- start start 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.868+0000 7f24efd0c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 0x7f24e81efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.868+0000 7f24efd0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e815d9c0 0x7f24e80b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.868+0000 7f24efd0c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e8150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.868+0000 7f24efd0c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f24e81efff0 con 0x7f24e80ca4a0 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.868+0000 7f24efd0c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f24e81f3760 con 0x7f24e815dda0 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.868+0000 7f24efd0c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24e815d7c0 con 0x7f24e815d9c0 2024-03-23T15:56:09.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.869+0000 7f24edaa8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 0x7f24e81efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:09.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.869+0000 7f24edaa8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 0x7f24e81efab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57686/0 (socket says 172.21.15.129:57686) 2024-03-23T15:56:09.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.869+0000 7f24edaa8640 1 -- 172.21.15.129:0/3397091431 learned_addr learned my addr 172.21.15.129:0/3397091431 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:09.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.871+0000 7f24ed2a7640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e815d9c0 0x7f24e80b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:09.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.872+0000 7f24ee2a9640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e8150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:09.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.873+0000 7f24edaa8640 1 -- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e815d9c0 msgr2=0x7f24e80b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:09.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.873+0000 7f24edaa8640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e815d9c0 0x7f24e80b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:09.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.873+0000 7f24edaa8640 1 -- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 msgr2=0x7f24e8150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:09.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.873+0000 7f24edaa8640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e8150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:09.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.873+0000 7f24edaa8640 1 -- 172.21.15.129:0/3397091431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e0009a70 con 0x7f24e80ca4a0 2024-03-23T15:56:09.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.875+0000 7f24edaa8640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 0x7f24e81efab0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c490 tx=0x7f24e400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:09.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.877+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4005ce0 con 0x7f24e80ca4a0 2024-03-23T15:56:09.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.877+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e815b6b0 con 0x7f24e80ca4a0 2024-03-23T15:56:09.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.877+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e815b3d0 con 0x7f24e80ca4a0 2024-03-23T15:56:09.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:09.879+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e8075740 con 0x7f24e80ca4a0 2024-03-23T15:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.131+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e4005e40 con 0x7f24e80ca4a0 2024-03-23T15:56:10.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.131+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4007ea0 con 0x7f24e80ca4a0 2024-03-23T15:56:10.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.132+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f24e4007310 con 0x7f24e80ca4a0 2024-03-23T15:56:10.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.132+0000 7f24de7fc640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f24cc06b840 0x7f24cc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.133+0000 7f24ed2a7640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f24cc06b840 0x7f24cc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.133+0000 7f24ed2a7640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f24cc06b840 0x7f24cc06dd00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24e81e71e0 tx=0x7f24d8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:10.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.146+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f24e4058720 con 0x7f24e80ca4a0 2024-03-23T15:56:10.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.146+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f24e40549c0 con 0x7f24e80ca4a0 2024-03-23T15:56:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.395+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24e8072c50 con 0x7f24e80ca4a0 2024-03-23T15:56:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.407+0000 7f24de7fc640 1 -- 172.21.15.129:0/3397091431 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f24e4017090 con 0x7f24e80ca4a0 2024-03-23T15:56:10.410 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:10.411 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:10.411 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f24cc06b840 msgr2=0x7f24cc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f24cc06b840 0x7f24cc06dd00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24e81e71e0 tx=0x7f24d8009190 comp rx=0 tx=0).stop 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 msgr2=0x7f24e81efab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 0x7f24e81efab0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c490 tx=0x7f24e400c950 comp rx=0 tx=0).stop 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 shutdown_connections 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f24cc06b840 0x7f24cc06dd00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e815dda0 0x7f24e8150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e815d9c0 0x7f24e80b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:10.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 --2- 172.21.15.129:0/3397091431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24e80ca4a0 0x7f24e81efab0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:10.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 >> 172.21.15.129:0/3397091431 conn(0x7f24e80cfb20 msgr2=0x7f24e80c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:10.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 shutdown_connections 2024-03-23T15:56:10.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:10.410+0000 7f24efd0c640 1 -- 172.21.15.129:0/3397091431 wait complete. 2024-03-23T15:56:11.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:11.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:12.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.129+0000 7ff763725640 1 Processor -- start 2024-03-23T15:56:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.130+0000 7ff763725640 1 -- start start 2024-03-23T15:56:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.130+0000 7ff763725640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff75c15a2e0 0x7ff75c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.130+0000 7ff763725640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff75c06c0b0 0x7ff75c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.130+0000 7ff763725640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c0cbcb0 con 0x7ff75c1544b0 2024-03-23T15:56:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.130+0000 7ff763725640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ff75c0c5be0 con 0x7ff75c06c0b0 2024-03-23T15:56:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.130+0000 7ff763725640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff75c15b280 con 0x7ff75c15a2e0 2024-03-23T15:56:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.131+0000 7ff761cc2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff75c06c0b0 0x7ff75c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.131+0000 7ff760cc0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff75c15a2e0 0x7ff75c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.131+0000 7ff761cc2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff75c06c0b0 0x7ff75c0c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47088/0 (socket says 172.21.15.129:47088) 2024-03-23T15:56:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.131+0000 7ff7614c1640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff75c1544b0 0x7ff75c0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:35726/0 (socket says 172.21.15.129:35726) 2024-03-23T15:56:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.131+0000 7ff761cc2640 1 -- 172.21.15.129:0/3488858875 learned_addr learned my addr 172.21.15.129:0/3488858875 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.135+0000 7ff761cc2640 1 -- 172.21.15.129:0/3488858875 >> v2:172.21.15.157:3300/0 conn(0x7ff75c15a2e0 msgr2=0x7ff75c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.135+0000 7ff761cc2640 1 --2- 172.21.15.129:0/3488858875 >> v2:172.21.15.157:3300/0 conn(0x7ff75c15a2e0 0x7ff75c06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.135+0000 7ff761cc2640 1 -- 172.21.15.129:0/3488858875 >> v1:172.21.15.157:6789/0 conn(0x7ff75c1544b0 legacy=0x7ff75c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:12.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.135+0000 7ff761cc2640 1 -- 172.21.15.129:0/3488858875 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c15a9e0 con 0x7ff75c06c0b0 2024-03-23T15:56:12.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.137+0000 7ff761cc2640 1 --2- 172.21.15.129:0/3488858875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c06c0b0 0x7ff75c0c6000 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff75c06ce60 tx=0x7ff75402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d6f49b854fa512 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.142+0000 7ff753fff640 1 -- 172.21.15.129:0/3488858875 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75402cbb0 con 0x7ff75c06c0b0 2024-03-23T15:56:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff753fff640 1 -- 172.21.15.129:0/3488858875 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75402cd10 con 0x7ff75c06c0b0 2024-03-23T15:56:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff753fff640 1 -- 172.21.15.129:0/3488858875 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754034610 con 0x7ff75c06c0b0 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 -- 172.21.15.129:0/3488858875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c06c0b0 msgr2=0x7ff75c0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 --2- 172.21.15.129:0/3488858875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c06c0b0 0x7ff75c0c6000 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff75c06ce60 tx=0x7ff75402c010 comp rx=0 tx=0).stop 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 -- 172.21.15.129:0/3488858875 shutdown_connections 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 --2- 172.21.15.129:0/3488858875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c06c0b0 0x7ff75c0c6000 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 --2- 172.21.15.129:0/3488858875 >> v2:172.21.15.157:3300/0 conn(0x7ff75c15a2e0 0x7ff75c06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 -- 172.21.15.129:0/3488858875 >> 172.21.15.129:0/3488858875 conn(0x7ff75c061b50 msgr2=0x7ff75c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 -- 172.21.15.129:0/3488858875 shutdown_connections 2024-03-23T15:56:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.143+0000 7ff763725640 1 -- 172.21.15.129:0/3488858875 wait complete. 2024-03-23T15:56:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 Processor -- start 2024-03-23T15:56:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 -- start start 2024-03-23T15:56:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff75c06c0b0 0x7ff75c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff75c1544b0 0x7ff75c1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 0x7ff75c0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c1f0840 con 0x7ff75c15a2e0 2024-03-23T15:56:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c1f4860 con 0x7ff75c1544b0 2024-03-23T15:56:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.145+0000 7ff763725640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff75c1f4b40 con 0x7ff75c06c0b0 2024-03-23T15:56:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.146+0000 7ff761cc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 0x7ff75c0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.146+0000 7ff760cc0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff75c1544b0 0x7ff75c1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.146+0000 7ff761cc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 0x7ff75c0b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47096/0 (socket says 172.21.15.129:47096) 2024-03-23T15:56:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.146+0000 7ff761cc2640 1 -- 172.21.15.129:0/1209398503 learned_addr learned my addr 172.21.15.129:0/1209398503 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.146+0000 7ff7614c1640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff75c06c0b0 0x7ff75c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.149+0000 7ff761cc2640 1 -- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff75c06c0b0 msgr2=0x7ff75c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.149+0000 7ff761cc2640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff75c06c0b0 0x7ff75c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.149+0000 7ff761cc2640 1 -- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff75c1544b0 msgr2=0x7ff75c1f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.149+0000 7ff761cc2640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff75c1544b0 0x7ff75c1f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.149+0000 7ff761cc2640 1 -- 172.21.15.129:0/1209398503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff754009b30 con 0x7ff75c15a2e0 2024-03-23T15:56:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.154+0000 7ff761cc2640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 0x7ff75c0b4090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff754036040 tx=0x7ff754004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:12.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.159+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754009510 con 0x7ff75c15a2e0 2024-03-23T15:56:12.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.159+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75403e450 con 0x7ff75c15a2e0 2024-03-23T15:56:12.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.159+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75403e800 con 0x7ff75c15a2e0 2024-03-23T15:56:12.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.159+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75c1f0b00 con 0x7ff75c15a2e0 2024-03-23T15:56:12.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.159+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75c1f10c0 con 0x7ff75c15a2e0 2024-03-23T15:56:12.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.160+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75c159280 con 0x7ff75c15a2e0 2024-03-23T15:56:12.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.184+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff75403e960 con 0x7ff75c15a2e0 2024-03-23T15:56:12.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.184+0000 7ff751ffb640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff74006b840 0x7ff74006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:12.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.184+0000 7ff7614c1640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff74006b840 0x7ff74006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:12.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.185+0000 7ff7614c1640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff74006b840 0x7ff74006dd00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff758007fd0 tx=0x7ff758007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:12.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.187+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff75407be00 con 0x7ff75c15a2e0 2024-03-23T15:56:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.446+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff7540780a0 con 0x7ff75c15a2e0 2024-03-23T15:56:12.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.664+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff75c075740 con 0x7ff75c15a2e0 2024-03-23T15:56:12.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.676+0000 7ff751ffb640 1 -- 172.21.15.129:0/1209398503 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff75407b760 con 0x7ff75c15a2e0 2024-03-23T15:56:12.680 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:12.680 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:12.680 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:12.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff74006b840 msgr2=0x7ff74006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:12.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff74006b840 0x7ff74006dd00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff758007fd0 tx=0x7ff758007e80 comp rx=0 tx=0).stop 2024-03-23T15:56:12.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 msgr2=0x7ff75c0b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:12.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 0x7ff75c0b4090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff754036040 tx=0x7ff754004a40 comp rx=0 tx=0).stop 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 shutdown_connections 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff75c15a2e0 0x7ff75c0b4090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff74006b840 0x7ff74006dd00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff75c1544b0 0x7ff75c1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 --2- 172.21.15.129:0/1209398503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff75c06c0b0 0x7ff75c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 >> 172.21.15.129:0/1209398503 conn(0x7ff75c061b50 msgr2=0x7ff75c1534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 shutdown_connections 2024-03-23T15:56:12.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:12.679+0000 7ff763725640 1 -- 172.21.15.129:0/1209398503 wait complete. 2024-03-23T15:56:13.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:13.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:14.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:14.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.397+0000 7ff2dd4fb640 1 Processor -- start 2024-03-23T15:56:14.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.397+0000 7ff2dd4fb640 1 -- start start 2024-03-23T15:56:14.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.399+0000 7ff2dd4fb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff2d80c8e90 0x7ff2d80c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.399+0000 7ff2dd4fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d80c3bd0 con 0x7ff2d8156f60 2024-03-23T15:56:14.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.399+0000 7ff2dd4fb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d80c3740 con 0x7ff2d80c7380 2024-03-23T15:56:14.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.399+0000 7ff2dd4fb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff2d80c57d0 con 0x7ff2d80c8e90 2024-03-23T15:56:14.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.400+0000 7ff2d77fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff2d8156f60 0x7ff2d8157370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:58020/0 (socket says 172.21.15.129:58020) 2024-03-23T15:56:14.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.400+0000 7ff2d77fe640 1 -- 172.21.15.129:0/2457078106 learned_addr learned my addr 172.21.15.129:0/2457078106 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.400+0000 7ff2d67fc640 1 --2- 172.21.15.129:0/2457078106 >> v2:172.21.15.157:3300/0 conn(0x7ff2d80c8e90 0x7ff2d80c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.402+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388433876 0 0) 0x7ff2d80c3bd0 con 0x7ff2d8156f60 2024-03-23T15:56:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.402+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c0005730 con 0x7ff2d8156f60 2024-03-23T15:56:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.403+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568961077 0 0) 0x7ff2d80c3740 con 0x7ff2d80c7380 2024-03-23T15:56:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.403+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d80c3bd0 con 0x7ff2d80c7380 2024-03-23T15:56:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.406+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2043781636 0 0) 0x7ff2c0005730 con 0x7ff2d8156f60 2024-03-23T15:56:14.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.406+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 >> v2:172.21.15.157:3300/0 conn(0x7ff2d80c8e90 msgr2=0x7ff2d80c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:14.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.406+0000 7ff2d5ffb640 1 --2- 172.21.15.129:0/2457078106 >> v2:172.21.15.157:3300/0 conn(0x7ff2d80c8e90 0x7ff2d80c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.406+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 >> v1:172.21.15.157:6789/0 conn(0x7ff2d80c7380 legacy=0x7ff2d80c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.406+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d80c80c0 con 0x7ff2d8156f60 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7ff2d00019f0 con 0x7ff2d8156f60 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 >> v1:172.21.15.129:6789/0 conn(0x7ff2d8156f60 legacy=0x7ff2d8157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2c0005d10 0x7ff2c00060f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2c00069d0 0x7ff2c0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2c000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d80c57d0 con 0x7ff2c0005d10 2024-03-23T15:56:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c000e180 con 0x7ff2c00069d0 2024-03-23T15:56:14.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.408+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c000e340 con 0x7ff2c0007390 2024-03-23T15:56:14.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.409+0000 7ff2d77fe640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2c000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.410+0000 7ff2d6ffd640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2c0005d10 0x7ff2c00060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.412+0000 7ff2d67fc640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2c00069d0 0x7ff2c0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.413+0000 7ff2d77fe640 1 -- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2c00069d0 msgr2=0x7ff2c0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:14.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.413+0000 7ff2d77fe640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2c00069d0 0x7ff2c0006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.413+0000 7ff2d77fe640 1 -- 172.21.15.129:0/2457078106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2c0005d10 msgr2=0x7ff2c00060f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:14.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.413+0000 7ff2d77fe640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2c0005d10 0x7ff2c00060f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.413+0000 7ff2d77fe640 1 -- 172.21.15.129:0/2457078106 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c0005730 con 0x7ff2c0007390 2024-03-23T15:56:14.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.414+0000 7ff2d77fe640 1 -- 172.21.15.129:0/2457078106 reap_dead start 2024-03-23T15:56:14.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.416+0000 7ff2d77fe640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2c000db90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0005ac0 tx=0x7ff2c0007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7032d8cd73a58f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:14.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.420+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d002d740 con 0x7ff2c0007390 2024-03-23T15:56:14.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.420+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d002d8a0 con 0x7ff2c0007390 2024-03-23T15:56:14.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.420+0000 7ff2d5ffb640 1 -- 172.21.15.129:0/2457078106 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d002dc30 con 0x7ff2c0007390 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 msgr2=0x7ff2c000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2c000db90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0005ac0 tx=0x7ff2c0007a00 comp rx=0 tx=0).stop 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/2457078106 shutdown_connections 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/2457078106 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2c000db90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/2457078106 >> 172.21.15.129:0/2457078106 conn(0x7ff2d80c8370 msgr2=0x7ff2d80c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/2457078106 shutdown_connections 2024-03-23T15:56:14.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.421+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/2457078106 wait complete. 2024-03-23T15:56:14.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 Processor -- start 2024-03-23T15:56:14.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 -- start start 2024-03-23T15:56:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2d8142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2d8156a30 0x7ff2d81540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d8145400 0x7ff2d8145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d8142a80 con 0x7ff2d8156a30 2024-03-23T15:56:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d8144c50 con 0x7ff2d8145400 2024-03-23T15:56:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.423+0000 7ff2dd4fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d80caa40 con 0x7ff2c0007390 2024-03-23T15:56:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.424+0000 7ff2d67fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2d8156a30 0x7ff2d81540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.424+0000 7ff2d77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d8145400 0x7ff2d8145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.424+0000 7ff2d6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2d8142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.424+0000 7ff2d67fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2d8156a30 0x7ff2d81540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47128/0 (socket says 172.21.15.129:47128) 2024-03-23T15:56:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.425+0000 7ff2d67fc640 1 -- 172.21.15.129:0/1354704551 learned_addr learned my addr 172.21.15.129:0/1354704551 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:14.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.428+0000 7ff2d6ffd640 1 -- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d8145400 msgr2=0x7ff2d8145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.428+0000 7ff2d6ffd640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d8145400 0x7ff2d8145880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.428+0000 7ff2d6ffd640 1 -- 172.21.15.129:0/1354704551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2d8156a30 msgr2=0x7ff2d81540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.428+0000 7ff2d6ffd640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2d8156a30 0x7ff2d81540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.428+0000 7ff2d6ffd640 1 -- 172.21.15.129:0/1354704551 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d80cb180 con 0x7ff2c0007390 2024-03-23T15:56:14.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.431+0000 7ff2d6ffd640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2d8142540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800f7a0 tx=0x7ff2c800fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:14.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.436+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8005110 con 0x7ff2c0007390 2024-03-23T15:56:14.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.436+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c8005290 con 0x7ff2c0007390 2024-03-23T15:56:14.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.436+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8007050 con 0x7ff2c0007390 2024-03-23T15:56:14.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.436+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d80cb460 con 0x7ff2c0007390 2024-03-23T15:56:14.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.436+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d81450e0 con 0x7ff2c0007390 2024-03-23T15:56:14.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.437+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d8075740 con 0x7ff2c0007390 2024-03-23T15:56:14.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.720+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff2c80071d0 con 0x7ff2c0007390 2024-03-23T15:56:14.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.720+0000 7ff2bf7fe640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2c406b860 0x7ff2c406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:14.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.720+0000 7ff2d67fc640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2c406b860 0x7ff2c406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:14.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.721+0000 7ff2d67fc640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2c406b860 0x7ff2c406dd20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8144850 tx=0x7ff2cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:14.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.724+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff2c8058850 con 0x7ff2c0007390 2024-03-23T15:56:14.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.735+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff2c8054af0 con 0x7ff2c0007390 2024-03-23T15:56:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.969+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2d8144d90 con 0x7ff2c0007390 2024-03-23T15:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.985+0000 7ff2bf7fe640 1 -- 172.21.15.129:0/1354704551 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff2c801a070 con 0x7ff2c0007390 2024-03-23T15:56:14.988 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:14.988 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2c406b860 msgr2=0x7ff2c406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2c406b860 0x7ff2c406dd20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8144850 tx=0x7ff2cc008040 comp rx=0 tx=0).stop 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 msgr2=0x7ff2d8142540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2d8142540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800f7a0 tx=0x7ff2c800fc60 comp rx=0 tx=0).stop 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 shutdown_connections 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff2c406b860 0x7ff2c406dd20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d8145400 0x7ff2d8145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2d8156a30 0x7ff2d81540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 --2- 172.21.15.129:0/1354704551 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2c0007390 0x7ff2d8142540 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:14.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 >> 172.21.15.129:0/1354704551 conn(0x7ff2d80c8370 msgr2=0x7ff2d806f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:14.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 shutdown_connections 2024-03-23T15:56:14.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:14.987+0000 7ff2dd4fb640 1 -- 172.21.15.129:0/1354704551 wait complete. 2024-03-23T15:56:15.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:15.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:16.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.707+0000 7f1ee9bc2640 1 Processor -- start 2024-03-23T15:56:16.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.707+0000 7f1ee9bc2640 1 -- start start 2024-03-23T15:56:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.708+0000 7f1ee9bc2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:16.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.708+0000 7f1ee9bc2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1ee40cc080 0x7f1ee415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.708+0000 7f1ee9bc2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ee406cae0 con 0x7f1ee40cbc60 2024-03-23T15:56:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.708+0000 7f1ee9bc2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee40cabc0 con 0x7f1ee40cc080 2024-03-23T15:56:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.708+0000 7f1ee9bc2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee40caea0 con 0x7f1ee406c030 2024-03-23T15:56:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.709+0000 7f1ee37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:16.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.709+0000 7f1ee2ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1ee40cbc60 0x7f1ee415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:35758/0 (socket says 172.21.15.129:35758) 2024-03-23T15:56:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.709+0000 7f1ee2ffd640 1 -- 172.21.15.129:0/254918391 learned_addr learned my addr 172.21.15.129:0/254918391 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:16.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.709+0000 7f1ee3fff640 1 --2- 172.21.15.129:0/254918391 >> v2:172.21.15.157:3300/0 conn(0x7f1ee40cc080 0x7f1ee415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.712+0000 7f1ee37fe640 1 -- 172.21.15.129:0/254918391 >> v2:172.21.15.157:3300/0 conn(0x7f1ee40cc080 msgr2=0x7f1ee415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.712+0000 7f1ee37fe640 1 --2- 172.21.15.129:0/254918391 >> v2:172.21.15.157:3300/0 conn(0x7f1ee40cc080 0x7f1ee415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.712+0000 7f1ee37fe640 1 -- 172.21.15.129:0/254918391 >> v1:172.21.15.157:6789/0 conn(0x7f1ee40cbc60 legacy=0x7f1ee415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:16.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.712+0000 7f1ee37fe640 1 -- 172.21.15.129:0/254918391 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee40cb180 con 0x7f1ee406c030 2024-03-23T15:56:16.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.714+0000 7f1ee37fe640 1 --2- 172.21.15.129:0/254918391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee406c410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40097a0 tx=0x7f1ed402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d715add4476a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.719+0000 7f1ee27fc640 1 -- 172.21.15.129:0/254918391 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed402e070 con 0x7f1ee406c030 2024-03-23T15:56:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.719+0000 7f1ee27fc640 1 -- 172.21.15.129:0/254918391 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed40315a0 con 0x7f1ee406c030 2024-03-23T15:56:16.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.719+0000 7f1ee27fc640 1 -- 172.21.15.129:0/254918391 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4031930 con 0x7f1ee406c030 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/254918391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 msgr2=0x7f1ee406c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/254918391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee406c410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40097a0 tx=0x7f1ed402b840 comp rx=0 tx=0).stop 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/254918391 shutdown_connections 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/254918391 >> v2:172.21.15.157:3300/0 conn(0x7f1ee40cc080 0x7f1ee415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/254918391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee406c410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/254918391 >> 172.21.15.129:0/254918391 conn(0x7f1ee40cfb20 msgr2=0x7f1ee40cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/254918391 shutdown_connections 2024-03-23T15:56:16.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.720+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/254918391 wait complete. 2024-03-23T15:56:16.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.722+0000 7f1ee9bc2640 1 Processor -- start 2024-03-23T15:56:16.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.722+0000 7f1ee9bc2640 1 -- start start 2024-03-23T15:56:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.723+0000 7f1ee9bc2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee41f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.723+0000 7f1ee9bc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ee40cbc60 0x7f1ee40b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.723+0000 7f1ee9bc2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 0x7f1ee40b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.723+0000 7f1ee9bc2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee40b5500 con 0x7f1ee40cbc60 2024-03-23T15:56:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.723+0000 7f1ee9bc2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee41f4380 con 0x7f1ee40cc080 2024-03-23T15:56:16.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.723+0000 7f1ee9bc2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee40b3fa0 con 0x7f1ee406c030 2024-03-23T15:56:16.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.724+0000 7f1ee37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee41f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:16.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.724+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 0x7f1ee40b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:16.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.724+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 0x7f1ee40b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:50328/0 (socket says 172.21.15.129:50328) 2024-03-23T15:56:16.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.724+0000 7f1ee3fff640 1 -- 172.21.15.129:0/449659747 learned_addr learned my addr 172.21.15.129:0/449659747 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:16.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee2ffd640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ee40cbc60 0x7f1ee40b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee3fff640 1 -- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 msgr2=0x7f1ee41f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee3fff640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee41f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee3fff640 1 -- 172.21.15.129:0/449659747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ee40cbc60 msgr2=0x7f1ee40b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee3fff640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ee40cbc60 0x7f1ee40b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee3fff640 1 -- 172.21.15.129:0/449659747 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed4009400 con 0x7f1ee40cc080 2024-03-23T15:56:16.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.728+0000 7f1ee2ffd640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ee40cbc60 0x7f1ee40b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T15:56:16.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.731+0000 7f1ee3fff640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 0x7f1ee40b4710 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1edc007ea0 tx=0x7f1edc007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.736+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc014620 con 0x7f1ee40cc080 2024-03-23T15:56:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.736+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1edc0147a0 con 0x7f1ee40cc080 2024-03-23T15:56:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.736+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc014b50 con 0x7f1ee40cc080 2024-03-23T15:56:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.736+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee41f03e0 con 0x7f1ee40cc080 2024-03-23T15:56:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.736+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee41f09a0 con 0x7f1ee40cc080 2024-03-23T15:56:16.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:16.737+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee4075740 con 0x7f1ee40cc080 2024-03-23T15:56:17.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.019+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1edc014cb0 con 0x7f1ee40cc080 2024-03-23T15:56:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.019+0000 7f1ecbfff640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1ed006b810 0x7f1ed006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.020+0000 7f1ee37fe640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1ed006b810 0x7f1ed006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.020+0000 7f1ee37fe640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1ed006b810 0x7f1ed006dcd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4036000 tx=0x7f1ed403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:17.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.024+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1edc00ee80 con 0x7f1ee40cc080 2024-03-23T15:56:17.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.035+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1edc0560b0 con 0x7f1ee40cc080 2024-03-23T15:56:17.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.323+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ee40d0a30 con 0x7f1ee40cc080 2024-03-23T15:56:17.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.336+0000 7f1ecbfff640 1 -- 172.21.15.129:0/449659747 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1edc059770 con 0x7f1ee40cc080 2024-03-23T15:56:17.339 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:17.339 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:17.339 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:17.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1ed006b810 msgr2=0x7f1ed006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:17.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1ed006b810 0x7f1ed006dcd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4036000 tx=0x7f1ed403d040 comp rx=0 tx=0).stop 2024-03-23T15:56:17.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 msgr2=0x7f1ee40b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:17.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 0x7f1ee40b4710 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1edc007ea0 tx=0x7f1edc007f90 comp rx=0 tx=0).stop 2024-03-23T15:56:17.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 shutdown_connections 2024-03-23T15:56:17.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1ed006b810 0x7f1ed006dcd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ee40cc080 0x7f1ee40b4710 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ee40cbc60 0x7f1ee40b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 --2- 172.21.15.129:0/449659747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ee406c030 0x7f1ee41f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 >> 172.21.15.129:0/449659747 conn(0x7f1ee40cfb20 msgr2=0x7f1ee40af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 shutdown_connections 2024-03-23T15:56:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:17.339+0000 7f1ee9bc2640 1 -- 172.21.15.129:0/449659747 wait complete. 2024-03-23T15:56:17.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:17.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:18.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:19.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.061+0000 7fc9e2d93640 1 Processor -- start 2024-03-23T15:56:19.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.061+0000 7fc9e2d93640 1 -- start start 2024-03-23T15:56:19.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.062+0000 7fc9e2d93640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc9dc06c030 0x7fc9dc06c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:19.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.062+0000 7fc9e2d93640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc9dc0cc080 0x7fc9dc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:19.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.062+0000 7fc9e2d93640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9dc06cae0 con 0x7fc9dc0cbc60 2024-03-23T15:56:19.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.062+0000 7fc9e2d93640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc9dc0cabc0 con 0x7fc9dc0cc080 2024-03-23T15:56:19.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.062+0000 7fc9e2d93640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc9dc0caea0 con 0x7fc9dc06c030 2024-03-23T15:56:19.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.063+0000 7fc9e1330640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc9dc0cc080 0x7fc9dc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:19.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.063+0000 7fc9e1330640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc9dc0cc080 0x7fc9dc15e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47140/0 (socket says 172.21.15.129:47140) 2024-03-23T15:56:19.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.063+0000 7fc9e1330640 1 -- 172.21.15.129:0/3137588450 learned_addr learned my addr 172.21.15.129:0/3137588450 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:19.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.063+0000 7fc9e0b2f640 1 --2- 172.21.15.129:0/3137588450 >> v2:172.21.15.157:3300/0 conn(0x7fc9dc06c030 0x7fc9dc06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:19.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.065+0000 7fc9e1330640 1 -- 172.21.15.129:0/3137588450 >> v2:172.21.15.157:3300/0 conn(0x7fc9dc06c030 msgr2=0x7fc9dc06c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.065+0000 7fc9e1330640 1 --2- 172.21.15.129:0/3137588450 >> v2:172.21.15.157:3300/0 conn(0x7fc9dc06c030 0x7fc9dc06c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.065+0000 7fc9e1330640 1 -- 172.21.15.129:0/3137588450 >> v1:172.21.15.157:6789/0 conn(0x7fc9dc0cbc60 legacy=0x7fc9dc15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:19.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.065+0000 7fc9e1330640 1 -- 172.21.15.129:0/3137588450 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc0cb180 con 0x7fc9dc0cc080 2024-03-23T15:56:19.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.067+0000 7fc9e1330640 1 --2- 172.21.15.129:0/3137588450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc0cc080 0x7fc9dc15e010 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0cf770 tx=0x7fc9d402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ff0230edd71606 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.071+0000 7fc9db7fe640 1 -- 172.21.15.129:0/3137588450 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d402cb20 con 0x7fc9dc0cc080 2024-03-23T15:56:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.071+0000 7fc9db7fe640 1 -- 172.21.15.129:0/3137588450 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d402cc80 con 0x7fc9dc0cc080 2024-03-23T15:56:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.071+0000 7fc9db7fe640 1 -- 172.21.15.129:0/3137588450 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d40343f0 con 0x7fc9dc0cc080 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3137588450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc0cc080 msgr2=0x7fc9dc15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3137588450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc0cc080 0x7fc9dc15e010 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0cf770 tx=0x7fc9d402bf80 comp rx=0 tx=0).stop 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3137588450 shutdown_connections 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3137588450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc0cc080 0x7fc9dc15e010 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3137588450 >> v2:172.21.15.157:3300/0 conn(0x7fc9dc06c030 0x7fc9dc06c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3137588450 >> 172.21.15.129:0/3137588450 conn(0x7fc9dc0cfb20 msgr2=0x7fc9dc0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:19.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3137588450 shutdown_connections 2024-03-23T15:56:19.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.072+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3137588450 wait complete. 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.073+0000 7fc9e2d93640 1 Processor -- start 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.073+0000 7fc9e2d93640 1 -- start start 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.074+0000 7fc9e2d93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 0x7fc9dc0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.074+0000 7fc9e2d93640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9dc0cbc60 0x7fc9dc157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.074+0000 7fc9e2d93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9dc0cc080 0x7fc9dc14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.074+0000 7fc9e2d93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc1efc70 con 0x7fc9dc06c030 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.074+0000 7fc9e2d93640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc150190 con 0x7fc9dc0cbc60 2024-03-23T15:56:19.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.074+0000 7fc9e2d93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9dc1efef0 con 0x7fc9dc0cc080 2024-03-23T15:56:19.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.075+0000 7fc9e0b2f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 0x7fc9dc0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:19.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.075+0000 7fc9e1330640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9dc0cc080 0x7fc9dc14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:19.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.075+0000 7fc9dbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9dc0cbc60 0x7fc9dc157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:19.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.075+0000 7fc9e0b2f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 0x7fc9dc0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47156/0 (socket says 172.21.15.129:47156) 2024-03-23T15:56:19.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.075+0000 7fc9e0b2f640 1 -- 172.21.15.129:0/3458309890 learned_addr learned my addr 172.21.15.129:0/3458309890 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:19.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.080+0000 7fc9e0b2f640 1 -- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9dc0cc080 msgr2=0x7fc9dc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:19.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.080+0000 7fc9e0b2f640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9dc0cc080 0x7fc9dc14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.080+0000 7fc9e0b2f640 1 -- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9dc0cbc60 msgr2=0x7fc9dc157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:19.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.080+0000 7fc9e0b2f640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9dc0cbc60 0x7fc9dc157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.080+0000 7fc9e0b2f640 1 -- 172.21.15.129:0/3458309890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d4009b30 con 0x7fc9dc06c030 2024-03-23T15:56:19.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.084+0000 7fc9e0b2f640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 0x7fc9dc0c5a90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000d800 tx=0x7fc9d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:19.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.088+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d000b6c0 con 0x7fc9dc06c030 2024-03-23T15:56:19.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.088+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d000b820 con 0x7fc9dc06c030 2024-03-23T15:56:19.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.088+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d000bbb0 con 0x7fc9dc06c030 2024-03-23T15:56:19.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.088+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9dc1f01d0 con 0x7fc9dc06c030 2024-03-23T15:56:19.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.088+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9dc1f0790 con 0x7fc9dc06c030 2024-03-23T15:56:19.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.089+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9dc1f0920 con 0x7fc9dc06c030 2024-03-23T15:56:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.123+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc9d000bd10 con 0x7fc9dc06c030 2024-03-23T15:56:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.124+0000 7fc9d97fa640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9c006b860 0x7fc9c006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:19.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.124+0000 7fc9dbfff640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9c006b860 0x7fc9c006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:19.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.125+0000 7fc9dbfff640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9c006b860 0x7fc9c006dd20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0c6510 tx=0x7fc9c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:19.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.389+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc9d0059970 con 0x7fc9dc06c030 2024-03-23T15:56:19.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.389+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc9d0055c10 con 0x7fc9dc06c030 2024-03-23T15:56:19.617 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:39.460 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.622 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:39.465 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.630 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:39.429 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.634 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:39.432 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.636 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:09:39.434 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.639 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:09:39.438 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.665+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9dc075740 con 0x7fc9dc06c030 2024-03-23T15:56:19.677 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:09:39.520 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.694 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:09:39.493 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.707 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:09:39.505 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:56:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.850+0000 7fc9d97fa640 1 -- 172.21.15.129:0/3458309890 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc9d00592d0 con 0x7fc9dc06c030 2024-03-23T15:56:19.853 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:19.853 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:19.853 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9c006b860 msgr2=0x7fc9c006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9c006b860 0x7fc9c006dd20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0c6510 tx=0x7fc9c8009040 comp rx=0 tx=0).stop 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 msgr2=0x7fc9dc0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 0x7fc9dc0c5a90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000d800 tx=0x7fc9d000dcc0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 shutdown_connections 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9dc0cc080 0x7fc9dc14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc9c006b860 0x7fc9c006dd20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9dc0cbc60 0x7fc9dc157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 --2- 172.21.15.129:0/3458309890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9dc06c030 0x7fc9dc0c5a90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:19.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 >> 172.21.15.129:0/3458309890 conn(0x7fc9dc0cfb20 msgr2=0x7fc9dc159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:19.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 shutdown_connections 2024-03-23T15:56:19.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:19.852+0000 7fc9e2d93640 1 -- 172.21.15.129:0/3458309890 wait complete. 2024-03-23T15:56:20.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:20.471 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:21.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:21.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.573+0000 7f4cf3b8a640 1 Processor -- start 2024-03-23T15:56:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.573+0000 7f4cf3b8a640 1 -- start start 2024-03-23T15:56:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.574+0000 7f4cf3b8a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4cec15dc60 0x7f4cec06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.574+0000 7f4cf3b8a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cec06d2c0 con 0x7f4cec06f570 2024-03-23T15:56:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.574+0000 7f4cf3b8a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cec15e740 con 0x7f4cec15e040 2024-03-23T15:56:21.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.574+0000 7f4cf3b8a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f4cec0ccd00 con 0x7f4cec15dc60 2024-03-23T15:56:21.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.574+0000 7f4cf1125640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4cec15e040 0x7f4cec0cdda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:32844/0 (socket says 172.21.15.129:32844) 2024-03-23T15:56:21.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.575+0000 7f4cf1125640 1 -- 172.21.15.129:0/3832416743 learned_addr learned my addr 172.21.15.129:0/3832416743 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:21.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.575+0000 7f4cf1926640 1 --2- 172.21.15.129:0/3832416743 >> v2:172.21.15.129:3300/0 conn(0x7f4cec15dc60 0x7f4cec06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:21.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.578+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042277528 0 0) 0x7f4cec15e740 con 0x7f4cec15e040 2024-03-23T15:56:21.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.578+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cd8003520 con 0x7f4cec15e040 2024-03-23T15:56:21.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.579+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417751957 0 0) 0x7f4cec06d2c0 con 0x7f4cec06f570 2024-03-23T15:56:21.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.579+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cec15e740 con 0x7f4cec06f570 2024-03-23T15:56:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.582+0000 7f4cf1926640 1 -- 172.21.15.129:0/3832416743 >> v1:172.21.15.157:6789/0 conn(0x7f4cec15e040 legacy=0x7f4cec0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.582+0000 7f4cf1926640 1 -- 172.21.15.129:0/3832416743 >> v1:172.21.15.129:6789/0 conn(0x7f4cec06f570 legacy=0x7f4cec0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.582+0000 7f4cf1926640 1 -- 172.21.15.129:0/3832416743 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cec0ccfe0 con 0x7f4cec15dc60 2024-03-23T15:56:21.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.586+0000 7f4cf1926640 1 --2- 172.21.15.129:0/3832416743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec15dc60 0x7f4cec06cc50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80097a0 tx=0x7f4ce802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbbaa65c071e40c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:21.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.590+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8031830 con 0x7f4cec15dc60 2024-03-23T15:56:21.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.590+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce8031990 con 0x7f4cec15dc60 2024-03-23T15:56:21.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf0924640 1 -- 172.21.15.129:0/3832416743 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8031d20 con 0x7f4cec15dc60 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/3832416743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec15dc60 msgr2=0x7f4cec06cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/3832416743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec15dc60 0x7f4cec06cc50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80097a0 tx=0x7f4ce802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/3832416743 shutdown_connections 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/3832416743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec15dc60 0x7f4cec06cc50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/3832416743 >> 172.21.15.129:0/3832416743 conn(0x7f4cec071300 msgr2=0x7f4cec0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/3832416743 shutdown_connections 2024-03-23T15:56:21.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.591+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/3832416743 wait complete. 2024-03-23T15:56:21.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 Processor -- start 2024-03-23T15:56:21.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 -- start start 2024-03-23T15:56:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 0x7f4cec14b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cec15dc60 0x7f4cec159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cec15e040 0x7f4cec1f0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec1f0ca0 con 0x7f4cec06f570 2024-03-23T15:56:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec14ce30 con 0x7f4cec15dc60 2024-03-23T15:56:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.593+0000 7f4cf3b8a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4cec15af70 con 0x7f4cec15e040 2024-03-23T15:56:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.594+0000 7f4cf1926640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 0x7f4cec14b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.594+0000 7f4cf1125640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cec15dc60 0x7f4cec159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.594+0000 7f4cf2127640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cec15e040 0x7f4cec1f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.594+0000 7f4cf1926640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 0x7f4cec14b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56474/0 (socket says 172.21.15.129:56474) 2024-03-23T15:56:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.594+0000 7f4cf1926640 1 -- 172.21.15.129:0/109269668 learned_addr learned my addr 172.21.15.129:0/109269668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.598+0000 7f4cf1926640 1 -- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cec15e040 msgr2=0x7f4cec1f0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.598+0000 7f4cf1926640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cec15e040 0x7f4cec1f0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.598+0000 7f4cf1926640 1 -- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cec15dc60 msgr2=0x7f4cec159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.598+0000 7f4cf1926640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cec15dc60 0x7f4cec159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.598+0000 7f4cf1926640 1 -- 172.21.15.129:0/109269668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce8009400 con 0x7f4cec06f570 2024-03-23T15:56:21.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.600+0000 7f4cf1926640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 0x7f4cec14b8d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4ce802bee0 tx=0x7f4ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.602+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce803a630 con 0x7f4cec06f570 2024-03-23T15:56:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.602+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cec15a9b0 con 0x7f4cec06f570 2024-03-23T15:56:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.603+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cec1592b0 con 0x7f4cec06f570 2024-03-23T15:56:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.603+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cec075740 con 0x7f4cec06f570 2024-03-23T15:56:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.605+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce8005990 con 0x7f4cec06f570 2024-03-23T15:56:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.605+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8005d00 con 0x7f4cec06f570 2024-03-23T15:56:21.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.860+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ce803a790 con 0x7f4cec06f570 2024-03-23T15:56:21.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.860+0000 7f4ce27fc640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4cd006b840 0x7f4cd006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:21.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.861+0000 7f4cf1125640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4cd006b840 0x7f4cd006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:21.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.861+0000 7f4cf1125640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4cd006b840 0x7f4cd006dd00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4cec153260 tx=0x7f4cdc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:21.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.873+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4ce807a8d0 con 0x7f4cec06f570 2024-03-23T15:56:21.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:21.874+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ce8076b70 con 0x7f4cec06f570 2024-03-23T15:56:22.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.095+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cec15b1f0 con 0x7f4cec06f570 2024-03-23T15:56:22.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.109+0000 7f4ce27fc640 1 -- 172.21.15.129:0/109269668 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4ce8048030 con 0x7f4cec06f570 2024-03-23T15:56:22.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:22.113 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:22.113 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:22.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4cd006b840 msgr2=0x7f4cd006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4cd006b840 0x7f4cd006dd00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4cec153260 tx=0x7f4cdc009040 comp rx=0 tx=0).stop 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 msgr2=0x7f4cec14b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 0x7f4cec14b8d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4ce802bee0 tx=0x7f4ce80046c0 comp rx=0 tx=0).stop 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 shutdown_connections 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cec15e040 0x7f4cec1f0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4cd006b840 0x7f4cd006dd00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cec15dc60 0x7f4cec159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 --2- 172.21.15.129:0/109269668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4cec06f570 0x7f4cec14b8d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 >> 172.21.15.129:0/109269668 conn(0x7f4cec071300 msgr2=0x7f4cec0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 shutdown_connections 2024-03-23T15:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:22.112+0000 7f4cf3b8a640 1 -- 172.21.15.129:0/109269668 wait complete. 2024-03-23T15:56:22.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:22.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:23.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:23.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.832+0000 7f25b1ace640 1 Processor -- start 2024-03-23T15:56:23.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.832+0000 7f25b1ace640 1 -- start start 2024-03-23T15:56:23.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.834+0000 7f25b1ace640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f25ac0ca580 0x7f25ac06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.834+0000 7f25b1ace640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25ac15d880 con 0x7f25ac154270 2024-03-23T15:56:23.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.834+0000 7f25b1ace640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25ac0ca9d0 con 0x7f25ac06c650 2024-03-23T15:56:23.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.834+0000 7f25b1ace640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f25ac0c4ae0 con 0x7f25ac0ca580 2024-03-23T15:56:23.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.834+0000 7f25ab7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f25ac154270 0x7f25ac15ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60556/0 (socket says 172.21.15.129:60556) 2024-03-23T15:56:23.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.834+0000 7f25ab7fe640 1 -- 172.21.15.129:0/1853099429 learned_addr learned my addr 172.21.15.129:0/1853099429 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:23.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.835+0000 7f25aaffd640 1 --2- 172.21.15.129:0/1853099429 >> v2:172.21.15.157:3300/0 conn(0x7f25ac0ca580 0x7f25ac06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:23.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.837+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 274908359 0 0) 0x7f25ac15d880 con 0x7f25ac154270 2024-03-23T15:56:23.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.837+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2594003520 con 0x7f25ac154270 2024-03-23T15:56:23.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1127957901 0 0) 0x7f2594003520 con 0x7f25ac154270 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 >> v2:172.21.15.157:3300/0 conn(0x7f25ac0ca580 msgr2=0x7f25ac06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 --2- 172.21.15.129:0/1853099429 >> v2:172.21.15.157:3300/0 conn(0x7f25ac0ca580 0x7f25ac06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 >> v1:172.21.15.157:6789/0 conn(0x7f25ac06c650 legacy=0x7f25ac0c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ac0cd5e0 con 0x7f25ac154270 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7f259c001670 con 0x7f25ac154270 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 >> v1:172.21.15.129:6789/0 conn(0x7f25ac154270 legacy=0x7f25ac15ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.840+0000 7f25aa7fc640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f2594003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.841+0000 7f25aa7fc640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25940047c0 0x7f2594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.841+0000 7f25aa7fc640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25940052a0 0x7f259400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.841+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25ac0c4ae0 con 0x7f25940047c0 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.841+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f259400c1a0 con 0x7f25940052a0 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.841+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f259400c360 con 0x7f25940039c0 2024-03-23T15:56:23.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.841+0000 7f25ab7fe640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f2594003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25aaffd640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25940047c0 0x7f2594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25ab7fe640 1 -- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25940052a0 msgr2=0x7f259400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25ab7fe640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25940052a0 0x7f259400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25ab7fe640 1 -- 172.21.15.129:0/1853099429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25940047c0 msgr2=0x7f2594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25ab7fe640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25940047c0 0x7f2594004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25ab7fe640 1 -- 172.21.15.129:0/1853099429 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2594003520 con 0x7f25940039c0 2024-03-23T15:56:23.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.844+0000 7f25abfff640 1 -- 172.21.15.129:0/1853099429 reap_dead start 2024-03-23T15:56:23.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.846+0000 7f25ab7fe640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f2594003de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f259c00c0e0 tx=0x7f25940057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b83c1419ffc6a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:23.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.849+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c005ad0 con 0x7f25940039c0 2024-03-23T15:56:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.851+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f259c005c50 con 0x7f25940039c0 2024-03-23T15:56:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.851+0000 7f25aa7fc640 1 -- 172.21.15.129:0/1853099429 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c02c7d0 con 0x7f25940039c0 2024-03-23T15:56:23.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 -- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 msgr2=0x7f2594003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f2594003de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f259c00c0e0 tx=0x7f25940057f0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 -- 172.21.15.129:0/1853099429 shutdown_connections 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 --2- 172.21.15.129:0/1853099429 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f2594003de0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 -- 172.21.15.129:0/1853099429 >> 172.21.15.129:0/1853099429 conn(0x7f25ac06d290 msgr2=0x7f25ac072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 -- 172.21.15.129:0/1853099429 shutdown_connections 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.852+0000 7f25b1ace640 1 -- 172.21.15.129:0/1853099429 wait complete. 2024-03-23T15:56:23.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.853+0000 7f25b1ace640 1 Processor -- start 2024-03-23T15:56:23.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.853+0000 7f25b1ace640 1 -- start start 2024-03-23T15:56:23.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25b1ace640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f25ac153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25b1ace640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25ac149910 0x7f25ac0b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25b1ace640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25ac0b3900 0x7f25ac1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:23.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25b1ace640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25ac157a90 con 0x7f25ac0b3900 2024-03-23T15:56:23.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25b1ace640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25ac15c870 con 0x7f25ac149910 2024-03-23T15:56:23.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25b1ace640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f25ac15c3e0 con 0x7f25940039c0 2024-03-23T15:56:23.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25ab7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f25ac153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:23.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.855+0000 7f25aaffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25ac149910 0x7f25ac0b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:23.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.856+0000 7f25ab7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f25ac153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44764/0 (socket says 172.21.15.129:44764) 2024-03-23T15:56:23.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.856+0000 7f25ab7fe640 1 -- 172.21.15.129:0/2385265601 learned_addr learned my addr 172.21.15.129:0/2385265601 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:23.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.856+0000 7f25abfff640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25ac0b3900 0x7f25ac1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.861+0000 7f25aaffd640 1 -- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 msgr2=0x7f25ac153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.861+0000 7f25aaffd640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f25ac153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.861+0000 7f25aaffd640 1 -- 172.21.15.129:0/2385265601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25ac0b3900 msgr2=0x7f25ac1524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.861+0000 7f25aaffd640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25ac0b3900 0x7f25ac1524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.861+0000 7f25aaffd640 1 -- 172.21.15.129:0/2385265601 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ac15b4f0 con 0x7f25ac149910 2024-03-23T15:56:23.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.865+0000 7f25aaffd640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25ac149910 0x7f25ac0b33c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f25a0010810 tx=0x7f25a0010cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:23.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.871+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0009590 con 0x7f25ac149910 2024-03-23T15:56:23.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.871+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a000eb60 con 0x7f25ac149910 2024-03-23T15:56:23.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.871+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25ac15b7d0 con 0x7f25ac149910 2024-03-23T15:56:23.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.871+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a001c400 con 0x7f25ac149910 2024-03-23T15:56:23.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.872+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25ac15bd90 con 0x7f25ac149910 2024-03-23T15:56:23.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:23.872+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25ac15c590 con 0x7f25ac149910 2024-03-23T15:56:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.140+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f25a001c560 con 0x7f25ac149910 2024-03-23T15:56:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.141+0000 7f2593fff640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f259806b840 0x7f259806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.141+0000 7f25ab7fe640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f259806b840 0x7f259806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:24.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.142+0000 7f25ab7fe640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f259806b840 0x7f259806dd00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f259c000c00 tx=0x7f259c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:24.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.145+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f25a0058ee0 con 0x7f25ac149910 2024-03-23T15:56:24.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.164+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f25a0055180 con 0x7f25ac149910 2024-03-23T15:56:24.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.384+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25ac15c590 con 0x7f25ac149910 2024-03-23T15:56:24.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.399+0000 7f2593fff640 1 -- 172.21.15.129:0/2385265601 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f25ac15c590 con 0x7f25ac149910 2024-03-23T15:56:24.402 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:24.402 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:24.402 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f259806b840 msgr2=0x7f259806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f259806b840 0x7f259806dd00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f259c000c00 tx=0x7f259c009030 comp rx=0 tx=0).stop 2024-03-23T15:56:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25ac149910 msgr2=0x7f25ac0b33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25ac149910 0x7f25ac0b33c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f25a0010810 tx=0x7f25a0010cd0 comp rx=0 tx=0).stop 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 shutdown_connections 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25ac0b3900 0x7f25ac1524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f259806b840 0x7f259806dd00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25ac149910 0x7f25ac0b33c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 --2- 172.21.15.129:0/2385265601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25940039c0 0x7f25ac153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.401+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 >> 172.21.15.129:0/2385265601 conn(0x7f25ac06d290 msgr2=0x7f25ac1585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.402+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 shutdown_connections 2024-03-23T15:56:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:24.402+0000 7f25b1ace640 1 -- 172.21.15.129:0/2385265601 wait complete. 2024-03-23T15:56:25.020 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:25.020 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:26.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.121+0000 7f095afdd640 1 Processor -- start 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.121+0000 7f095afdd640 1 -- start start 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.122+0000 7f095afdd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f09541544b0 0x7f095415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.122+0000 7f095afdd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09540caa00 0x7f095406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.122+0000 7f095afdd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f095406c0b0 0x7f09540c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.122+0000 7f095afdd640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f09540c5be0 con 0x7f09540caa00 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.122+0000 7f095afdd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f09540cbcb0 con 0x7f09541544b0 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.122+0000 7f095afdd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f095415b280 con 0x7f095406c0b0 2024-03-23T15:56:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.123+0000 7f0953fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09540caa00 0x7f095406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.123+0000 7f0953fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f09540caa00 0x7f095406bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56498/0 (socket says 172.21.15.129:56498) 2024-03-23T15:56:26.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.123+0000 7f0953fff640 1 -- 172.21.15.129:0/3036043655 learned_addr learned my addr 172.21.15.129:0/3036043655 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:26.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.123+0000 7f095957a640 1 --2- 172.21.15.129:0/3036043655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f095406c0b0 0x7f09540c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.124+0000 7f0958d79640 1 --2- 172.21.15.129:0/3036043655 >> v2:172.21.15.157:3300/0 conn(0x7f09541544b0 0x7f095415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.125+0000 7f0953fff640 1 -- 172.21.15.129:0/3036043655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f095406c0b0 msgr2=0x7f09540c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.125+0000 7f0953fff640 1 --2- 172.21.15.129:0/3036043655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f095406c0b0 0x7f09540c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.125+0000 7f0953fff640 1 -- 172.21.15.129:0/3036043655 >> v2:172.21.15.157:3300/0 conn(0x7f09541544b0 msgr2=0x7f095415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.125+0000 7f0953fff640 1 --2- 172.21.15.129:0/3036043655 >> v2:172.21.15.157:3300/0 conn(0x7f09541544b0 0x7f095415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.125+0000 7f0953fff640 1 -- 172.21.15.129:0/3036043655 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095415a9e0 con 0x7f09540caa00 2024-03-23T15:56:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.127+0000 7f0953fff640 1 --2- 172.21.15.129:0/3036043655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09540caa00 0x7f095406bb70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f095406cd70 tx=0x7f094002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183a1c54bd4c0a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:26.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.131+0000 7f09537fe640 1 -- 172.21.15.129:0/3036043655 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940030860 con 0x7f09540caa00 2024-03-23T15:56:26.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.131+0000 7f09537fe640 1 -- 172.21.15.129:0/3036043655 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09400309c0 con 0x7f09540caa00 2024-03-23T15:56:26.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.131+0000 7f09537fe640 1 -- 172.21.15.129:0/3036043655 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940030d50 con 0x7f09540caa00 2024-03-23T15:56:26.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 -- 172.21.15.129:0/3036043655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09540caa00 msgr2=0x7f095406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 --2- 172.21.15.129:0/3036043655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09540caa00 0x7f095406bb70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f095406cd70 tx=0x7f094002b9a0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 -- 172.21.15.129:0/3036043655 shutdown_connections 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 --2- 172.21.15.129:0/3036043655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f095406c0b0 0x7f09540c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 --2- 172.21.15.129:0/3036043655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09540caa00 0x7f095406bb70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 --2- 172.21.15.129:0/3036043655 >> v2:172.21.15.157:3300/0 conn(0x7f09541544b0 0x7f095415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 -- 172.21.15.129:0/3036043655 >> 172.21.15.129:0/3036043655 conn(0x7f0954061b50 msgr2=0x7f0954061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 -- 172.21.15.129:0/3036043655 shutdown_connections 2024-03-23T15:56:26.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.132+0000 7f095afdd640 1 -- 172.21.15.129:0/3036043655 wait complete. 2024-03-23T15:56:26.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.133+0000 7f095afdd640 1 Processor -- start 2024-03-23T15:56:26.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.133+0000 7f095afdd640 1 -- start start 2024-03-23T15:56:26.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095afdd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f095406c0b0 0x7f095414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095afdd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09540caa00 0x7f095414a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095afdd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 0x7f0954156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095afdd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09540d0630 con 0x7f09541544b0 2024-03-23T15:56:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095afdd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f095414ba70 con 0x7f095406c0b0 2024-03-23T15:56:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095afdd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0954157b20 con 0x7f09540caa00 2024-03-23T15:56:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.134+0000 7f095957a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 0x7f0954156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.135+0000 7f0953fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09540caa00 0x7f095414a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.135+0000 7f0958d79640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f095406c0b0 0x7f095414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.135+0000 7f095957a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 0x7f0954156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56506/0 (socket says 172.21.15.129:56506) 2024-03-23T15:56:26.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.135+0000 7f095957a640 1 -- 172.21.15.129:0/1338420608 learned_addr learned my addr 172.21.15.129:0/1338420608 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:26.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.138+0000 7f095957a640 1 -- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09540caa00 msgr2=0x7f095414a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:26.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.138+0000 7f095957a640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09540caa00 0x7f095414a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.138+0000 7f095957a640 1 -- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f095406c0b0 msgr2=0x7f095414b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.138+0000 7f095957a640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f095406c0b0 0x7f095414b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.138+0000 7f095957a640 1 -- 172.21.15.129:0/1338420608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09400094d0 con 0x7f09541544b0 2024-03-23T15:56:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.141+0000 7f095957a640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 0x7f0954156550 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f094c00eed0 tx=0x7f094c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:26.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.145+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c010040 con 0x7f09541544b0 2024-03-23T15:56:26.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.145+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094c014400 con 0x7f09541544b0 2024-03-23T15:56:26.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.145+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09540cfab0 con 0x7f09541544b0 2024-03-23T15:56:26.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.145+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c009ea0 con 0x7f09541544b0 2024-03-23T15:56:26.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.145+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09540d0070 con 0x7f09541544b0 2024-03-23T15:56:26.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.146+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09540b3450 con 0x7f09541544b0 2024-03-23T15:56:26.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.423+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f094c0147b0 con 0x7f09541544b0 2024-03-23T15:56:26.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.423+0000 7f09517fa640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f093806b840 0x7f093806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:26.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.424+0000 7f0958d79640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f093806b840 0x7f093806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:26.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.424+0000 7f0958d79640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f093806b840 0x7f093806dd00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f09480079a0 tx=0x7f0948008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:26.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.446+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f094c059900 con 0x7f09541544b0 2024-03-23T15:56:26.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.446+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f094c055ba0 con 0x7f09541544b0 2024-03-23T15:56:26.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.714+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0954075740 con 0x7f09541544b0 2024-03-23T15:56:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.727+0000 7f09517fa640 1 -- 172.21.15.129:0/1338420608 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f094c018020 con 0x7f09541544b0 2024-03-23T15:56:26.730 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:26.730 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f093806b840 msgr2=0x7f093806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f093806b840 0x7f093806dd00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f09480079a0 tx=0x7f0948008040 comp rx=0 tx=0).stop 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 msgr2=0x7f0954156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 0x7f0954156550 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f094c00eed0 tx=0x7f094c00b480 comp rx=0 tx=0).stop 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 shutdown_connections 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09541544b0 0x7f0954156550 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09540caa00 0x7f095414a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f093806b840 0x7f093806dd00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 --2- 172.21.15.129:0/1338420608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f095406c0b0 0x7f095414b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 >> 172.21.15.129:0/1338420608 conn(0x7f0954061b50 msgr2=0x7f09540b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 shutdown_connections 2024-03-23T15:56:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:26.729+0000 7f095afdd640 1 -- 172.21.15.129:0/1338420608 wait complete. 2024-03-23T15:56:27.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:27.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:28.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:28.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.451+0000 7f4a9b384640 1 Processor -- start 2024-03-23T15:56:28.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.452+0000 7f4a9b384640 1 -- start start 2024-03-23T15:56:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.453+0000 7f4a9b384640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 0x7f4a9415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.453+0000 7f4a9b384640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f4a940cef40 0x7f4a9415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.453+0000 7f4a9b384640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a9415b290 con 0x7f4a940ccd60 2024-03-23T15:56:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.453+0000 7f4a9b384640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f4a940ce800 con 0x7f4a940cef40 2024-03-23T15:56:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.453+0000 7f4a9b384640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4a94154a70 con 0x7f4a940ceb00 2024-03-23T15:56:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.454+0000 7f4a9891f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 0x7f4a9415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.454+0000 7f4a9891f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 0x7f4a9415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44802/0 (socket says 172.21.15.129:44802) 2024-03-23T15:56:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.454+0000 7f4a9891f640 1 -- 172.21.15.129:0/3077955786 learned_addr learned my addr 172.21.15.129:0/3077955786 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.454+0000 7f4a99921640 1 --2- 172.21.15.129:0/3077955786 >> v2:172.21.15.129:3300/0 conn(0x7f4a940cef40 0x7f4a9415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:28.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.457+0000 7f4a9891f640 1 -- 172.21.15.129:0/3077955786 >> v2:172.21.15.129:3300/0 conn(0x7f4a940cef40 msgr2=0x7f4a9415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:28.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.457+0000 7f4a9891f640 1 --2- 172.21.15.129:0/3077955786 >> v2:172.21.15.129:3300/0 conn(0x7f4a940cef40 0x7f4a9415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:28.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.457+0000 7f4a9891f640 1 -- 172.21.15.129:0/3077955786 >> v1:172.21.15.129:6789/0 conn(0x7f4a940ccd60 legacy=0x7f4a9406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:28.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.457+0000 7f4a9891f640 1 -- 172.21.15.129:0/3077955786 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a940cd950 con 0x7f4a940ceb00 2024-03-23T15:56:28.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.460+0000 7f4a9891f640 1 --2- 172.21.15.129:0/3077955786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 0x7f4a9415ad50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009870 tx=0x7f4a8402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9248bbb381157dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:28.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.464+0000 7f4a8bfff640 1 -- 172.21.15.129:0/3077955786 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8402f840 con 0x7f4a940ceb00 2024-03-23T15:56:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.464+0000 7f4a8bfff640 1 -- 172.21.15.129:0/3077955786 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8402f9a0 con 0x7f4a940ceb00 2024-03-23T15:56:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.464+0000 7f4a8bfff640 1 -- 172.21.15.129:0/3077955786 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8402fd30 con 0x7f4a940ceb00 2024-03-23T15:56:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 -- 172.21.15.129:0/3077955786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 msgr2=0x7f4a9415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3077955786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 0x7f4a9415ad50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009870 tx=0x7f4a8402ba20 comp rx=0 tx=0).stop 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 -- 172.21.15.129:0/3077955786 shutdown_connections 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3077955786 >> v2:172.21.15.129:3300/0 conn(0x7f4a940cef40 0x7f4a9415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3077955786 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940ceb00 0x7f4a9415ad50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 -- 172.21.15.129:0/3077955786 >> 172.21.15.129:0/3077955786 conn(0x7f4a940b1840 msgr2=0x7f4a940b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 -- 172.21.15.129:0/3077955786 shutdown_connections 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.465+0000 7f4a9b384640 1 -- 172.21.15.129:0/3077955786 wait complete. 2024-03-23T15:56:28.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.466+0000 7f4a9b384640 1 Processor -- start 2024-03-23T15:56:28.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.466+0000 7f4a9b384640 1 -- start start 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a9b384640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 0x7f4a941f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a9b384640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a940ceb00 0x7f4a941f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a9b384640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940cef40 0x7f4a941521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a9b384640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941509a0 con 0x7f4a940ccd60 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a9b384640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941f50d0 con 0x7f4a940ceb00 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a9b384640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4a94151750 con 0x7f4a940cef40 2024-03-23T15:56:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.467+0000 7f4a99120640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 0x7f4a941f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.468+0000 7f4a99921640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940cef40 0x7f4a941521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.468+0000 7f4a9891f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a940ceb00 0x7f4a941f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.468+0000 7f4a99120640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 0x7f4a941f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56538/0 (socket says 172.21.15.129:56538) 2024-03-23T15:56:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.468+0000 7f4a99120640 1 -- 172.21.15.129:0/3359434366 learned_addr learned my addr 172.21.15.129:0/3359434366 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:28.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.471+0000 7f4a99120640 1 -- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940cef40 msgr2=0x7f4a941521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:28.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.471+0000 7f4a99120640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940cef40 0x7f4a941521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:28.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.471+0000 7f4a99120640 1 -- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a940ceb00 msgr2=0x7f4a941f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:28.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.471+0000 7f4a99120640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a940ceb00 0x7f4a941f15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:28.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.471+0000 7f4a99120640 1 -- 172.21.15.129:0/3359434366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a840094d0 con 0x7f4a940ccd60 2024-03-23T15:56:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.473+0000 7f4a99120640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 0x7f4a941f4b90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000c400 tx=0x7f4a9000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.477+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a900052d0 con 0x7f4a940ccd60 2024-03-23T15:56:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.477+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a90005450 con 0x7f4a940ccd60 2024-03-23T15:56:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.477+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a94150b90 con 0x7f4a940ccd60 2024-03-23T15:56:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.477+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90007260 con 0x7f4a940ccd60 2024-03-23T15:56:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.478+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a94151470 con 0x7f4a940ccd60 2024-03-23T15:56:28.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.479+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9404f530 con 0x7f4a940ccd60 2024-03-23T15:56:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.501+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4a900073c0 con 0x7f4a940ccd60 2024-03-23T15:56:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.502+0000 7f4a89ffb640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4a7806b860 0x7f4a7806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:28.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.502+0000 7f4a9891f640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4a7806b860 0x7f4a7806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:28.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.503+0000 7f4a9891f640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4a7806b860 0x7f4a7806dd20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4a84036000 tx=0x7f4a84008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:28.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.795+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4a90007670 con 0x7f4a940ccd60 2024-03-23T15:56:28.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:28.795+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4a90054700 con 0x7f4a940ccd60 2024-03-23T15:56:29.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.075+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a940686f0 con 0x7f4a940ccd60 2024-03-23T15:56:29.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.127+0000 7f4a89ffb640 1 -- 172.21.15.129:0/3359434366 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4a90020070 con 0x7f4a940ccd60 2024-03-23T15:56:29.130 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:29.130 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:29.130 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:29.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4a7806b860 msgr2=0x7f4a7806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:29.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4a7806b860 0x7f4a7806dd20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4a84036000 tx=0x7f4a84008030 comp rx=0 tx=0).stop 2024-03-23T15:56:29.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 msgr2=0x7f4a941f4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:29.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 0x7f4a941f4b90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000c400 tx=0x7f4a9000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 shutdown_connections 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a940cef40 0x7f4a941521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4a7806b860 0x7f4a7806dd20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a940ceb00 0x7f4a941f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 --2- 172.21.15.129:0/3359434366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a940ccd60 0x7f4a941f4b90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 >> 172.21.15.129:0/3359434366 conn(0x7f4a940b1840 msgr2=0x7f4a94154230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 shutdown_connections 2024-03-23T15:56:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:29.129+0000 7f4a9b384640 1 -- 172.21.15.129:0/3359434366 wait complete. 2024-03-23T15:56:29.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:29.747 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:30.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:30.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.850+0000 7ff22ebac640 1 Processor -- start 2024-03-23T15:56:30.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.850+0000 7ff22ebac640 1 -- start start 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22ebac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff2281544b0 0x7ff22815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22ebac640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2280cbcb0 con 0x7ff2280c6000 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22ebac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22815b280 con 0x7ff2280caa00 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22ebac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff2280c5be0 con 0x7ff2281544b0 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22d149640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff2280c6000 0x7ff228164c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44630/0 (socket says 172.21.15.129:44630) 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22d149640 1 -- 172.21.15.129:0/2924697969 learned_addr learned my addr 172.21.15.129:0/2924697969 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:30.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.851+0000 7ff22c948640 1 --2- 172.21.15.129:0/2924697969 >> v2:172.21.15.157:3300/0 conn(0x7ff2281544b0 0x7ff22815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:30.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.854+0000 7ff2277fe640 1 -- 172.21.15.129:0/2924697969 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 44425335 0 0) 0x7ff2280cbcb0 con 0x7ff2280c6000 2024-03-23T15:56:30.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.854+0000 7ff2277fe640 1 -- 172.21.15.129:0/2924697969 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff218003520 con 0x7ff2280c6000 2024-03-23T15:56:30.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.856+0000 7ff22c948640 1 -- 172.21.15.129:0/2924697969 >> v1:172.21.15.157:6789/0 conn(0x7ff2280caa00 legacy=0x7ff22806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:30.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.856+0000 7ff22c948640 1 -- 172.21.15.129:0/2924697969 >> v1:172.21.15.129:6789/0 conn(0x7ff2280c6000 legacy=0x7ff228164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:30.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.856+0000 7ff22c948640 1 -- 172.21.15.129:0/2924697969 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22815a9e0 con 0x7ff2281544b0 2024-03-23T15:56:30.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.862+0000 7ff22c948640 1 --2- 172.21.15.129:0/2924697969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2281544b0 0x7ff22815a2e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc0fa4da1de4e8e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:30.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.868+0000 7ff2277fe640 1 -- 172.21.15.129:0/2924697969 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c02f810 con 0x7ff2281544b0 2024-03-23T15:56:30.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.868+0000 7ff2277fe640 1 -- 172.21.15.129:0/2924697969 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff21c02f970 con 0x7ff2281544b0 2024-03-23T15:56:30.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.868+0000 7ff2277fe640 1 -- 172.21.15.129:0/2924697969 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c02fd00 con 0x7ff2281544b0 2024-03-23T15:56:30.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 -- 172.21.15.129:0/2924697969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2281544b0 msgr2=0x7ff22815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 --2- 172.21.15.129:0/2924697969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2281544b0 0x7ff22815a2e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c02b9f0 comp rx=0 tx=0).stop 2024-03-23T15:56:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 -- 172.21.15.129:0/2924697969 shutdown_connections 2024-03-23T15:56:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 --2- 172.21.15.129:0/2924697969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2281544b0 0x7ff22815a2e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 -- 172.21.15.129:0/2924697969 >> 172.21.15.129:0/2924697969 conn(0x7ff228061b50 msgr2=0x7ff228061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 -- 172.21.15.129:0/2924697969 shutdown_connections 2024-03-23T15:56:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.869+0000 7ff22ebac640 1 -- 172.21.15.129:0/2924697969 wait complete. 2024-03-23T15:56:30.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.870+0000 7ff22ebac640 1 Processor -- start 2024-03-23T15:56:30.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.870+0000 7ff22ebac640 1 -- start start 2024-03-23T15:56:30.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22ebac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 0x7ff2280cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:30.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22ebac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2280caa00 0x7ff228147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:30.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22ebac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2281544b0 0x7ff228147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:30.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22ebac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2280d0880 con 0x7ff2280c6000 2024-03-23T15:56:30.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22ebac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff2280cfcd0 con 0x7ff2280caa00 2024-03-23T15:56:30.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22ebac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff2281550a0 con 0x7ff2281544b0 2024-03-23T15:56:30.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.871+0000 7ff22c948640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 0x7ff2280cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:30.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.872+0000 7ff22d149640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2281544b0 0x7ff228147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:30.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.872+0000 7ff22c948640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 0x7ff2280cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58032/0 (socket says 172.21.15.129:58032) 2024-03-23T15:56:30.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.872+0000 7ff22c948640 1 -- 172.21.15.129:0/1694843067 learned_addr learned my addr 172.21.15.129:0/1694843067 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:30.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.873+0000 7ff227fff640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2280caa00 0x7ff228147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:30.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.875+0000 7ff22c948640 1 -- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2281544b0 msgr2=0x7ff228147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:30.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.875+0000 7ff22c948640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2281544b0 0x7ff228147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:30.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.875+0000 7ff22c948640 1 -- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2280caa00 msgr2=0x7ff228147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:30.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.875+0000 7ff22c948640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2280caa00 0x7ff228147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:30.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.875+0000 7ff22c948640 1 -- 172.21.15.129:0/1694843067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c009400 con 0x7ff2280c6000 2024-03-23T15:56:30.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.877+0000 7ff22c948640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 0x7ff2280cf450 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff21c02c000 tx=0x7ff21c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.882+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c03c040 con 0x7ff2280c6000 2024-03-23T15:56:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.882+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff21c0378f0 con 0x7ff2280c6000 2024-03-23T15:56:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.882+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c037ca0 con 0x7ff2280c6000 2024-03-23T15:56:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.882+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2280cfee0 con 0x7ff2280c6000 2024-03-23T15:56:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.882+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2280d04a0 con 0x7ff2280c6000 2024-03-23T15:56:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:30.883+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2281f33f0 con 0x7ff2280c6000 2024-03-23T15:56:31.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.134+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff21c04b3b0 con 0x7ff2280c6000 2024-03-23T15:56:31.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.134+0000 7ff2257fa640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff20c06b840 0x7ff20c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:31.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.135+0000 7ff227fff640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff20c06b840 0x7ff20c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:31.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.135+0000 7ff227fff640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff20c06b840 0x7ff20c06dd00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff2280b5040 tx=0x7ff214009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:31.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.150+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff21c07adf0 con 0x7ff2280c6000 2024-03-23T15:56:31.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.150+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff21c077090 con 0x7ff2280c6000 2024-03-23T15:56:31.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.466+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff228075740 con 0x7ff2280c6000 2024-03-23T15:56:31.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.487+0000 7ff2257fa640 1 -- 172.21.15.129:0/1694843067 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff21c07a750 con 0x7ff2280c6000 2024-03-23T15:56:31.489 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:31.490 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:31.490 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff20c06b840 msgr2=0x7ff20c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff20c06b840 0x7ff20c06dd00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff2280b5040 tx=0x7ff214009040 comp rx=0 tx=0).stop 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 msgr2=0x7ff2280cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 0x7ff2280cf450 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff21c02c000 tx=0x7ff21c004730 comp rx=0 tx=0).stop 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 shutdown_connections 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff2281544b0 0x7ff228147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff20c06b840 0x7ff20c06dd00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2280caa00 0x7ff228147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 --2- 172.21.15.129:0/1694843067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2280c6000 0x7ff2280cf450 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 >> 172.21.15.129:0/1694843067 conn(0x7ff228061b50 msgr2=0x7ff228158130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 shutdown_connections 2024-03-23T15:56:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:31.489+0000 7ff22ebac640 1 -- 172.21.15.129:0/1694843067 wait complete. 2024-03-23T15:56:32.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:32.108 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:33.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:33.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.210+0000 7fe1402af640 1 Processor -- start 2024-03-23T15:56:33.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.210+0000 7fe1402af640 1 -- start start 2024-03-23T15:56:33.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.211+0000 7fe1402af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe1380ccd60 0x7fe13815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:33.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.211+0000 7fe1402af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380cef40 0x7fe13815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:33.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.211+0000 7fe1402af640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe13815b290 con 0x7fe1380ceb00 2024-03-23T15:56:33.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.211+0000 7fe1402af640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe1380ce800 con 0x7fe1380ccd60 2024-03-23T15:56:33.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.211+0000 7fe1402af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe138154a70 con 0x7fe1380cef40 2024-03-23T15:56:33.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.212+0000 7fe13e04b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe1380ccd60 0x7fe13815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:33.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.212+0000 7fe13e84c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380cef40 0x7fe13815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:33.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.212+0000 7fe13e04b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe1380ccd60 0x7fe13815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:56170/0 (socket says 172.21.15.129:56170) 2024-03-23T15:56:33.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.212+0000 7fe13e04b640 1 -- 172.21.15.129:0/3809912739 learned_addr learned my addr 172.21.15.129:0/3809912739 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:33.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.215+0000 7fe13e84c640 1 -- 172.21.15.129:0/3809912739 >> v2:172.21.15.157:3300/0 conn(0x7fe1380ccd60 msgr2=0x7fe13815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:33.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.215+0000 7fe13e84c640 1 --2- 172.21.15.129:0/3809912739 >> v2:172.21.15.157:3300/0 conn(0x7fe1380ccd60 0x7fe13815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.215+0000 7fe13e84c640 1 -- 172.21.15.129:0/3809912739 >> v1:172.21.15.157:6789/0 conn(0x7fe1380ceb00 legacy=0x7fe13806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:33.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.215+0000 7fe13e84c640 1 -- 172.21.15.129:0/3809912739 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1380cd950 con 0x7fe1380cef40 2024-03-23T15:56:33.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.218+0000 7fe13e84c640 1 --2- 172.21.15.129:0/3809912739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380cef40 0x7fe13815b8f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe130009d90 tx=0x7fe13002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=168824abae1ef67c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:33.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.221+0000 7fe13d049640 1 -- 172.21.15.129:0/3809912739 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13002cce0 con 0x7fe1380cef40 2024-03-23T15:56:33.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.223+0000 7fe13d049640 1 -- 172.21.15.129:0/3809912739 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13002ce40 con 0x7fe1380cef40 2024-03-23T15:56:33.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.223+0000 7fe13d049640 1 -- 172.21.15.129:0/3809912739 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130034660 con 0x7fe1380cef40 2024-03-23T15:56:33.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.224+0000 7fe1402af640 1 -- 172.21.15.129:0/3809912739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380cef40 msgr2=0x7fe13815b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:33.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.224+0000 7fe1402af640 1 --2- 172.21.15.129:0/3809912739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380cef40 0x7fe13815b8f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe130009d90 tx=0x7fe13002bf30 comp rx=0 tx=0).stop 2024-03-23T15:56:33.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.225+0000 7fe1402af640 1 -- 172.21.15.129:0/3809912739 shutdown_connections 2024-03-23T15:56:33.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.225+0000 7fe1402af640 1 --2- 172.21.15.129:0/3809912739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380cef40 0x7fe13815b8f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.225+0000 7fe1402af640 1 --2- 172.21.15.129:0/3809912739 >> v2:172.21.15.157:3300/0 conn(0x7fe1380ccd60 0x7fe13815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.225+0000 7fe1402af640 1 -- 172.21.15.129:0/3809912739 >> 172.21.15.129:0/3809912739 conn(0x7fe1380b1840 msgr2=0x7fe1380b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:33.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.225+0000 7fe1402af640 1 -- 172.21.15.129:0/3809912739 shutdown_connections 2024-03-23T15:56:33.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.225+0000 7fe1402af640 1 -- 172.21.15.129:0/3809912739 wait complete. 2024-03-23T15:56:33.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.226+0000 7fe1402af640 1 Processor -- start 2024-03-23T15:56:33.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.226+0000 7fe1402af640 1 -- start start 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe1402af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 0x7fe138150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe1402af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380ceb00 0x7fe13815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe1402af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1380cef40 0x7fe13815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe1402af640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1380c93b0 con 0x7fe1380ccd60 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe1402af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe1381f4300 con 0x7fe1380cef40 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe1402af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe1381f3d60 con 0x7fe1380ceb00 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe13e04b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 0x7fe138150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe13e84c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1380cef40 0x7fe13815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.228+0000 7fe13d84a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380ceb00 0x7fe13815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.229+0000 7fe13e04b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 0x7fe138150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58034/0 (socket says 172.21.15.129:58034) 2024-03-23T15:56:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.229+0000 7fe13e04b640 1 -- 172.21.15.129:0/377626184 learned_addr learned my addr 172.21.15.129:0/377626184 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:33.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.232+0000 7fe13e04b640 1 -- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380ceb00 msgr2=0x7fe13815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:33.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.232+0000 7fe13e04b640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380ceb00 0x7fe13815bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.232+0000 7fe13e04b640 1 -- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1380cef40 msgr2=0x7fe13815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:33.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.232+0000 7fe13e04b640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1380cef40 0x7fe13815c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.232+0000 7fe13e04b640 1 -- 172.21.15.129:0/377626184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe130009a70 con 0x7fe1380ccd60 2024-03-23T15:56:33.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.235+0000 7fe13e04b640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 0x7fe138150c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe134009fd0 tx=0x7fe13400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:33.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.239+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13400c940 con 0x7fe1380ccd60 2024-03-23T15:56:33.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.239+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13400caa0 con 0x7fe1380ccd60 2024-03-23T15:56:33.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.239+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13400ce30 con 0x7fe1380ccd60 2024-03-23T15:56:33.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.239+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1380c6dd0 con 0x7fe1380ccd60 2024-03-23T15:56:33.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.239+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13815caf0 con 0x7fe1380ccd60 2024-03-23T15:56:33.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.240+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13804f530 con 0x7fe1380ccd60 2024-03-23T15:56:33.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.498+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe134013470 con 0x7fe1380ccd60 2024-03-23T15:56:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.498+0000 7fe12effd640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe11c06b860 0x7fe11c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.499+0000 7fe13d84a640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe11c06b860 0x7fe11c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.499+0000 7fe13d84a640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe11c06b860 0x7fe11c06dd20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe1380b40a0 tx=0x7fe128009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:33.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.512+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe134014e40 con 0x7fe1380ccd60 2024-03-23T15:56:33.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.512+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe134055530 con 0x7fe1380ccd60 2024-03-23T15:56:33.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.791+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe13806db50 con 0x7fe1380ccd60 2024-03-23T15:56:33.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.804+0000 7fe12effd640 1 -- 172.21.15.129:0/377626184 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe134017020 con 0x7fe1380ccd60 2024-03-23T15:56:33.808 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:33.808 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:33.808 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:33.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe11c06b860 msgr2=0x7fe11c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:33.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe11c06b860 0x7fe11c06dd20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe1380b40a0 tx=0x7fe128009040 comp rx=0 tx=0).stop 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 msgr2=0x7fe138150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 0x7fe138150c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe134009fd0 tx=0x7fe13400ed40 comp rx=0 tx=0).stop 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 shutdown_connections 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe11c06b860 0x7fe11c06dd20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.806+0000 7fe1402af640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1380cef40 0x7fe13815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.807+0000 7fe1402af640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1380ceb00 0x7fe13815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.807+0000 7fe1402af640 1 --2- 172.21.15.129:0/377626184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1380ccd60 0x7fe138150c00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:33.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.807+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 >> 172.21.15.129:0/377626184 conn(0x7fe1380b1840 msgr2=0x7fe1380b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:33.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.807+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 shutdown_connections 2024-03-23T15:56:33.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:33.807+0000 7fe1402af640 1 -- 172.21.15.129:0/377626184 wait complete. 2024-03-23T15:56:34.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:34.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:35.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:35.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.528+0000 7fd1e106b640 1 Processor -- start 2024-03-23T15:56:35.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.528+0000 7fd1e106b640 1 -- start start 2024-03-23T15:56:35.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1e106b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:35.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1e106b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd1dc0c6000 0x7fd1dc0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:35.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1e106b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1dc0cbcb0 con 0x7fd1dc0caa00 2024-03-23T15:56:35.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1e106b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd1dc0c5be0 con 0x7fd1dc0c6000 2024-03-23T15:56:35.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1e106b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1dc0c69b0 con 0x7fd1dc1544b0 2024-03-23T15:56:35.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1da59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fd1dc0caa00 0x7fd1dc06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44644/0 (socket says 172.21.15.129:44644) 2024-03-23T15:56:35.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1da59c640 1 -- 172.21.15.129:0/3568034976 learned_addr learned my addr 172.21.15.129:0/3568034976 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:35.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1db59e640 1 --2- 172.21.15.129:0/3568034976 >> v2:172.21.15.157:3300/0 conn(0x7fd1dc0c6000 0x7fd1dc0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:35.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.530+0000 7fd1dad9d640 1 --2- 172.21.15.129:0/3568034976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:35.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.533+0000 7fd1dad9d640 1 -- 172.21.15.129:0/3568034976 >> v2:172.21.15.157:3300/0 conn(0x7fd1dc0c6000 msgr2=0x7fd1dc0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:35.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.533+0000 7fd1dad9d640 1 --2- 172.21.15.129:0/3568034976 >> v2:172.21.15.157:3300/0 conn(0x7fd1dc0c6000 0x7fd1dc0c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:35.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.533+0000 7fd1dad9d640 1 -- 172.21.15.129:0/3568034976 >> v1:172.21.15.129:6789/0 conn(0x7fd1dc0caa00 legacy=0x7fd1dc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:35.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.533+0000 7fd1dad9d640 1 -- 172.21.15.129:0/3568034976 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1dc15b280 con 0x7fd1dc1544b0 2024-03-23T15:56:35.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.535+0000 7fd1dad9d640 1 --2- 172.21.15.129:0/3568034976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc15a2e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0097a0 tx=0x7fd1cc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=234746572b9981a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:35.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.540+0000 7fd1d9d9b640 1 -- 172.21.15.129:0/3568034976 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc030610 con 0x7fd1dc1544b0 2024-03-23T15:56:35.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.540+0000 7fd1d9d9b640 1 -- 172.21.15.129:0/3568034976 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1cc030770 con 0x7fd1dc1544b0 2024-03-23T15:56:35.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.540+0000 7fd1d9d9b640 1 -- 172.21.15.129:0/3568034976 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc030b00 con 0x7fd1dc1544b0 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 -- 172.21.15.129:0/3568034976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 msgr2=0x7fd1dc15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 --2- 172.21.15.129:0/3568034976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc15a2e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0097a0 tx=0x7fd1cc02b960 comp rx=0 tx=0).stop 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 -- 172.21.15.129:0/3568034976 shutdown_connections 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 --2- 172.21.15.129:0/3568034976 >> v2:172.21.15.157:3300/0 conn(0x7fd1dc0c6000 0x7fd1dc0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 --2- 172.21.15.129:0/3568034976 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc15a2e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 -- 172.21.15.129:0/3568034976 >> 172.21.15.129:0/3568034976 conn(0x7fd1dc061b50 msgr2=0x7fd1dc061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 -- 172.21.15.129:0/3568034976 shutdown_connections 2024-03-23T15:56:35.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.541+0000 7fd1e106b640 1 -- 172.21.15.129:0/3568034976 wait complete. 2024-03-23T15:56:35.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.543+0000 7fd1e106b640 1 Processor -- start 2024-03-23T15:56:35.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.543+0000 7fd1e106b640 1 -- start start 2024-03-23T15:56:35.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.544+0000 7fd1e106b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 0x7fd1dc1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:35.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.544+0000 7fd1e106b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1dc0caa00 0x7fd1dc145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:35.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.544+0000 7fd1e106b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc0ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:35.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.544+0000 7fd1e106b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc155c70 con 0x7fd1dc0c6000 2024-03-23T15:56:35.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.544+0000 7fd1e106b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1dc156c00 con 0x7fd1dc0caa00 2024-03-23T15:56:35.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.544+0000 7fd1e106b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1dc144c90 con 0x7fd1dc1544b0 2024-03-23T15:56:35.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.545+0000 7fd1dad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 0x7fd1dc1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:35.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.545+0000 7fd1db59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc0ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:35.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.545+0000 7fd1da59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1dc0caa00 0x7fd1dc145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:35.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.545+0000 7fd1dad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 0x7fd1dc1555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58048/0 (socket says 172.21.15.129:58048) 2024-03-23T15:56:35.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.545+0000 7fd1dad9d640 1 -- 172.21.15.129:0/2812232530 learned_addr learned my addr 172.21.15.129:0/2812232530 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.548+0000 7fd1dad9d640 1 -- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 msgr2=0x7fd1dc0ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.548+0000 7fd1dad9d640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc0ceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.548+0000 7fd1dad9d640 1 -- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1dc0caa00 msgr2=0x7fd1dc145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.548+0000 7fd1dad9d640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1dc0caa00 0x7fd1dc145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.548+0000 7fd1dad9d640 1 -- 172.21.15.129:0/2812232530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc009400 con 0x7fd1dc0c6000 2024-03-23T15:56:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.551+0000 7fd1dad9d640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 0x7fd1dc1555e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc02bf70 tx=0x7fd1cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:35.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.556+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc03e070 con 0x7fd1dc0c6000 2024-03-23T15:56:35.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.556+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1cc030e10 con 0x7fd1dc0c6000 2024-03-23T15:56:35.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.556+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1dc0d0700 con 0x7fd1dc0c6000 2024-03-23T15:56:35.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.556+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc039c50 con 0x7fd1dc0c6000 2024-03-23T15:56:35.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.556+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1dc1562d0 con 0x7fd1dc0c6000 2024-03-23T15:56:35.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.557+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1dc0ca770 con 0x7fd1dc0c6000 2024-03-23T15:56:35.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.579+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd1cc055430 con 0x7fd1dc0c6000 2024-03-23T15:56:35.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.580+0000 7fd1c37fe640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd1c806b860 0x7fd1c806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:35.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.580+0000 7fd1da59c640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd1c806b860 0x7fd1c806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:35.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.581+0000 7fd1da59c640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd1c806b860 0x7fd1c806dd20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc1469b0 tx=0x7fd1d0007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:35.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.836+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd1cc07b530 con 0x7fd1dc0c6000 2024-03-23T15:56:35.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:35.836+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd1cc0777d0 con 0x7fd1dc0c6000 2024-03-23T15:56:36.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.064+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1dc0d0700 con 0x7fd1dc0c6000 2024-03-23T15:56:36.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.078+0000 7fd1c37fe640 1 -- 172.21.15.129:0/2812232530 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd1dc0d0700 con 0x7fd1dc0c6000 2024-03-23T15:56:36.081 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:36.081 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:36.081 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd1c806b860 msgr2=0x7fd1c806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd1c806b860 0x7fd1c806dd20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc1469b0 tx=0x7fd1d0007e80 comp rx=0 tx=0).stop 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 msgr2=0x7fd1dc1555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 0x7fd1dc1555e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc02bf70 tx=0x7fd1cc0046c0 comp rx=0 tx=0).stop 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 shutdown_connections 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1dc1544b0 0x7fd1dc0ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd1c806b860 0x7fd1c806dd20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:36.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1dc0caa00 0x7fd1dc145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:36.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 --2- 172.21.15.129:0/2812232530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1dc0c6000 0x7fd1dc1555e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:36.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 >> 172.21.15.129:0/2812232530 conn(0x7fd1dc061b50 msgr2=0x7fd1dc0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:36.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 shutdown_connections 2024-03-23T15:56:36.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:36.080+0000 7fd1e106b640 1 -- 172.21.15.129:0/2812232530 wait complete. 2024-03-23T15:56:36.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:36.698 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:37.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.798+0000 7fed8f93e640 1 Processor -- start 2024-03-23T15:56:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.798+0000 7fed8f93e640 1 -- start start 2024-03-23T15:56:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8f93e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fed880ceb00 0x7fed8815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8f93e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed8815b290 con 0x7fed880ccd60 2024-03-23T15:56:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8f93e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880ce800 con 0x7fed880cef40 2024-03-23T15:56:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8f93e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fed88154a70 con 0x7fed880ceb00 2024-03-23T15:56:37.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8d6da640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fed880ccd60 0x7fed8806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44658/0 (socket says 172.21.15.129:44658) 2024-03-23T15:56:37.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8d6da640 1 -- 172.21.15.129:0/92735977 learned_addr learned my addr 172.21.15.129:0/92735977 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:37.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.799+0000 7fed8ced9640 1 --2- 172.21.15.129:0/92735977 >> v2:172.21.15.157:3300/0 conn(0x7fed880ceb00 0x7fed8815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:37.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.802+0000 7fed7ffff640 1 -- 172.21.15.129:0/92735977 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130546929 0 0) 0x7fed84001110 con 0x7fed880ccd60 2024-03-23T15:56:37.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.802+0000 7fed7ffff640 1 -- 172.21.15.129:0/92735977 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed74003520 con 0x7fed880ccd60 2024-03-23T15:56:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.803+0000 7fed8ced9640 1 -- 172.21.15.129:0/92735977 >> v1:172.21.15.157:6789/0 conn(0x7fed880cef40 legacy=0x7fed8815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.803+0000 7fed8ced9640 1 -- 172.21.15.129:0/92735977 >> v1:172.21.15.129:6789/0 conn(0x7fed880ccd60 legacy=0x7fed8806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.803+0000 7fed8ced9640 1 -- 172.21.15.129:0/92735977 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed880cd950 con 0x7fed880ceb00 2024-03-23T15:56:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.808+0000 7fed8ced9640 1 --2- 172.21.15.129:0/92735977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880ceb00 0x7fed8815ad50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fed780097a0 tx=0x7fed7802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad9fc5f7727f9f54 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.812+0000 7fed7ffff640 1 -- 172.21.15.129:0/92735977 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7802f810 con 0x7fed880ceb00 2024-03-23T15:56:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.812+0000 7fed7ffff640 1 -- 172.21.15.129:0/92735977 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed7802f970 con 0x7fed880ceb00 2024-03-23T15:56:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.812+0000 7fed7ffff640 1 -- 172.21.15.129:0/92735977 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7802fd00 con 0x7fed880ceb00 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 -- 172.21.15.129:0/92735977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880ceb00 msgr2=0x7fed8815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 --2- 172.21.15.129:0/92735977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880ceb00 0x7fed8815ad50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fed780097a0 tx=0x7fed7802b9f0 comp rx=0 tx=0).stop 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 -- 172.21.15.129:0/92735977 shutdown_connections 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 --2- 172.21.15.129:0/92735977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880ceb00 0x7fed8815ad50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 -- 172.21.15.129:0/92735977 >> 172.21.15.129:0/92735977 conn(0x7fed880b1840 msgr2=0x7fed880b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 -- 172.21.15.129:0/92735977 shutdown_connections 2024-03-23T15:56:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.813+0000 7fed8f93e640 1 -- 172.21.15.129:0/92735977 wait complete. 2024-03-23T15:56:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.814+0000 7fed8f93e640 1 Processor -- start 2024-03-23T15:56:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 -- start start 2024-03-23T15:56:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed880ccd60 0x7fed88157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 0x7fed880c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880cef40 0x7fed880c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed880c7320 con 0x7fed880ceb00 2024-03-23T15:56:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fed88158470 con 0x7fed880cef40 2024-03-23T15:56:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.815+0000 7fed8f93e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fed880c96c0 con 0x7fed880ccd60 2024-03-23T15:56:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.816+0000 7fed8ced9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 0x7fed880c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.816+0000 7fed8d6da640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed880ccd60 0x7fed88157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.816+0000 7fed8dedb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880cef40 0x7fed880c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.816+0000 7fed8ced9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 0x7fed880c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58060/0 (socket says 172.21.15.129:58060) 2024-03-23T15:56:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.816+0000 7fed8ced9640 1 -- 172.21.15.129:0/4203241846 learned_addr learned my addr 172.21.15.129:0/4203241846 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.819+0000 7fed8ced9640 1 -- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed880ccd60 msgr2=0x7fed88157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.819+0000 7fed8ced9640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed880ccd60 0x7fed88157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.819+0000 7fed8ced9640 1 -- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880cef40 msgr2=0x7fed880c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.819+0000 7fed8ced9640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880cef40 0x7fed880c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.819+0000 7fed8ced9640 1 -- 172.21.15.129:0/4203241846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed78009400 con 0x7fed880ceb00 2024-03-23T15:56:37.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.822+0000 7fed8ced9640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 0x7fed880c8050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fed7802c000 tx=0x7fed78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.825+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7803c040 con 0x7fed880ceb00 2024-03-23T15:56:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.825+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed780378f0 con 0x7fed880ceb00 2024-03-23T15:56:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.825+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed880c99a0 con 0x7fed880ceb00 2024-03-23T15:56:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.825+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78037ca0 con 0x7fed880ceb00 2024-03-23T15:56:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.825+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed880c7850 con 0x7fed880ceb00 2024-03-23T15:56:37.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.827+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed8804f530 con 0x7fed880ceb00 2024-03-23T15:56:37.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.844+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fed780423f0 con 0x7fed880ceb00 2024-03-23T15:56:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.845+0000 7fed7dffb640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed6c06b860 0x7fed6c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.845+0000 7fed8d6da640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed6c06b860 0x7fed6c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.845+0000 7fed8d6da640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed6c06b860 0x7fed6c06dd20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fed84001270 tx=0x7fed84007bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:37.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:37.853+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fed7807aa60 con 0x7fed880ceb00 2024-03-23T15:56:38.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.104+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fed78076d00 con 0x7fed880ceb00 2024-03-23T15:56:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.325+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed88075740 con 0x7fed880ceb00 2024-03-23T15:56:38.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.338+0000 7fed7dffb640 1 -- 172.21.15.129:0/4203241846 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fed78047070 con 0x7fed880ceb00 2024-03-23T15:56:38.341 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:38.341 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:38.341 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed6c06b860 msgr2=0x7fed6c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed6c06b860 0x7fed6c06dd20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fed84001270 tx=0x7fed84007bb0 comp rx=0 tx=0).stop 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 msgr2=0x7fed880c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 0x7fed880c8050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fed7802c000 tx=0x7fed78004730 comp rx=0 tx=0).stop 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 shutdown_connections 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fed6c06b860 0x7fed6c06dd20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed880cef40 0x7fed880c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed880ceb00 0x7fed880c8050 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 --2- 172.21.15.129:0/4203241846 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed880ccd60 0x7fed88157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 >> 172.21.15.129:0/4203241846 conn(0x7fed880b1840 msgr2=0x7fed880b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 shutdown_connections 2024-03-23T15:56:38.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:38.341+0000 7fed8f93e640 1 -- 172.21.15.129:0/4203241846 wait complete. 2024-03-23T15:56:38.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:56:38.379 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:56:38.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:38.960 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:39.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:40.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.060+0000 7f134e73a640 1 Processor -- start 2024-03-23T15:56:40.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.060+0000 7f134e73a640 1 -- start start 2024-03-23T15:56:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f134e73a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f13480ccd60 0x7f134815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f134e73a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f13480cef40 0x7f134815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f134e73a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134815b290 con 0x7f13480ceb00 2024-03-23T15:56:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f134e73a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f13480ce800 con 0x7f13480ccd60 2024-03-23T15:56:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f134e73a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1348154a70 con 0x7f13480cef40 2024-03-23T15:56:40.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f1347fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f13480ccd60 0x7f134815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:40.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f134ccd7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f13480cef40 0x7f134815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:40.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f1347fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f13480ccd60 0x7f134815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58072/0 (socket says 172.21.15.129:58072) 2024-03-23T15:56:40.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.062+0000 7f1347fff640 1 -- 172.21.15.129:0/3327787543 learned_addr learned my addr 172.21.15.129:0/3327787543 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.065+0000 7f1347fff640 1 -- 172.21.15.129:0/3327787543 >> v2:172.21.15.157:3300/0 conn(0x7f13480cef40 msgr2=0x7f134815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.065+0000 7f1347fff640 1 --2- 172.21.15.129:0/3327787543 >> v2:172.21.15.157:3300/0 conn(0x7f13480cef40 0x7f134815b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.065+0000 7f1347fff640 1 -- 172.21.15.129:0/3327787543 >> v1:172.21.15.157:6789/0 conn(0x7f13480ceb00 legacy=0x7f134806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:40.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.065+0000 7f1347fff640 1 -- 172.21.15.129:0/3327787543 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13480cd950 con 0x7f13480ccd60 2024-03-23T15:56:40.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.067+0000 7f1347fff640 1 --2- 172.21.15.129:0/3327787543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f134815ad50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f13480cf660 tx=0x7f133802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06ff14d28d8d904 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.070+0000 7f1346ffd640 1 -- 172.21.15.129:0/3327787543 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338030860 con 0x7f13480ccd60 2024-03-23T15:56:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.070+0000 7f1346ffd640 1 -- 172.21.15.129:0/3327787543 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13380309c0 con 0x7f13480ccd60 2024-03-23T15:56:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.070+0000 7f1346ffd640 1 -- 172.21.15.129:0/3327787543 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338030d50 con 0x7f13480ccd60 2024-03-23T15:56:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.071+0000 7f134e73a640 1 -- 172.21.15.129:0/3327787543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 msgr2=0x7f134815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.071+0000 7f134e73a640 1 --2- 172.21.15.129:0/3327787543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f134815ad50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f13480cf660 tx=0x7f133802b9a0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.072+0000 7f134e73a640 1 -- 172.21.15.129:0/3327787543 shutdown_connections 2024-03-23T15:56:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.072+0000 7f134e73a640 1 --2- 172.21.15.129:0/3327787543 >> v2:172.21.15.157:3300/0 conn(0x7f13480cef40 0x7f134815b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.072+0000 7f134e73a640 1 --2- 172.21.15.129:0/3327787543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f134815ad50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.072+0000 7f134e73a640 1 -- 172.21.15.129:0/3327787543 >> 172.21.15.129:0/3327787543 conn(0x7f13480b1840 msgr2=0x7f13480b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.072+0000 7f134e73a640 1 -- 172.21.15.129:0/3327787543 shutdown_connections 2024-03-23T15:56:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.072+0000 7f134e73a640 1 -- 172.21.15.129:0/3327787543 wait complete. 2024-03-23T15:56:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.073+0000 7f134e73a640 1 Processor -- start 2024-03-23T15:56:40.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.073+0000 7f134e73a640 1 -- start start 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f134e73a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f13480c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f134e73a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13480ceb00 0x7f13480cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f134e73a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13480cef40 0x7f13480c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f134e73a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13480c8950 con 0x7f13480ccd60 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f134e73a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13480c8ab0 con 0x7f13480ceb00 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f134e73a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f13480c9bb0 con 0x7f13480cef40 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f1347fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f13480c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.074+0000 7f13477fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13480ceb00 0x7f13480cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.075+0000 7f1347fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f13480c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58074/0 (socket says 172.21.15.129:58074) 2024-03-23T15:56:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.075+0000 7f1347fff640 1 -- 172.21.15.129:0/2868814246 learned_addr learned my addr 172.21.15.129:0/2868814246 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.075+0000 7f134ccd7640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13480cef40 0x7f13480c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:40.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.078+0000 7f1347fff640 1 -- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13480cef40 msgr2=0x7f13480c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:40.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.078+0000 7f1347fff640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13480cef40 0x7f13480c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.078+0000 7f1347fff640 1 -- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13480ceb00 msgr2=0x7f13480cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.078+0000 7f1347fff640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13480ceb00 0x7f13480cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.078+0000 7f1347fff640 1 -- 172.21.15.129:0/2868814246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13380094d0 con 0x7f13480ccd60 2024-03-23T15:56:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.080+0000 7f1347fff640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f13480c8410 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1338000c00 tx=0x7f1338004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.084+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338039a90 con 0x7f13480ccd60 2024-03-23T15:56:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.084+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1338039bf0 con 0x7f13480ccd60 2024-03-23T15:56:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.084+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13480c8cc0 con 0x7f13480ccd60 2024-03-23T15:56:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.084+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13380433f0 con 0x7f13480ccd60 2024-03-23T15:56:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.084+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13480c9280 con 0x7f13480ccd60 2024-03-23T15:56:40.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.085+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134804f530 con 0x7f13480ccd60 2024-03-23T15:56:40.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.389+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f133804c3b0 con 0x7f13480ccd60 2024-03-23T15:56:40.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.389+0000 7f1344ff9640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f133406b890 0x7f133406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:40.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.390+0000 7f13477fe640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f133406b890 0x7f133406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:40.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.390+0000 7f13477fe640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f133406b890 0x7f133406dd50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f13480cca10 tx=0x7f133c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:40.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.402+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f133807b410 con 0x7f13480ccd60 2024-03-23T15:56:40.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.403+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f13380776b0 con 0x7f13480ccd60 2024-03-23T15:56:40.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.653+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13480686f0 con 0x7f13480ccd60 2024-03-23T15:56:40.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.691+0000 7f1344ff9640 1 -- 172.21.15.129:0/2868814246 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1338034070 con 0x7f13480ccd60 2024-03-23T15:56:40.693 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:40.694 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:40.694 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f133406b890 msgr2=0x7f133406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f133406b890 0x7f133406dd50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f13480cca10 tx=0x7f133c009060 comp rx=0 tx=0).stop 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 msgr2=0x7f13480c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f13480c8410 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1338000c00 tx=0x7f1338004730 comp rx=0 tx=0).stop 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 shutdown_connections 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13480cef40 0x7f13480c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f133406b890 0x7f133406dd50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13480ceb00 0x7f13480cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 --2- 172.21.15.129:0/2868814246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13480ccd60 0x7f13480c8410 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 >> 172.21.15.129:0/2868814246 conn(0x7f13480b1840 msgr2=0x7f13480cbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:40.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 shutdown_connections 2024-03-23T15:56:40.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:40.694+0000 7f134e73a640 1 -- 172.21.15.129:0/2868814246 wait complete. 2024-03-23T15:56:41.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:41.312 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:42.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:42.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.414+0000 7fa0d0747640 1 Processor -- start 2024-03-23T15:56:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.414+0000 7fa0d0747640 1 -- start start 2024-03-23T15:56:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.415+0000 7fa0d0747640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa0c80ceb00 0x7fa0c815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.415+0000 7fa0d0747640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c815b290 con 0x7fa0c80cef40 2024-03-23T15:56:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.415+0000 7fa0d0747640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c80ce800 con 0x7fa0c80ccd60 2024-03-23T15:56:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.415+0000 7fa0d0747640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c8154a70 con 0x7fa0c80ceb00 2024-03-23T15:56:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.416+0000 7fa0ce4e3640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa0c80ccd60 0x7fa0c806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:48928/0 (socket says 172.21.15.129:48928) 2024-03-23T15:56:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.416+0000 7fa0ce4e3640 1 -- 172.21.15.129:0/1515337814 learned_addr learned my addr 172.21.15.129:0/1515337814 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:42.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.416+0000 7fa0cdce2640 1 --2- 172.21.15.129:0/1515337814 >> v2:172.21.15.129:3300/0 conn(0x7fa0c80ceb00 0x7fa0c815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:42.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.419+0000 7fa0cd4e1640 1 -- 172.21.15.129:0/1515337814 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2541480050 0 0) 0x7fa0c80ce800 con 0x7fa0c80ccd60 2024-03-23T15:56:42.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.419+0000 7fa0cd4e1640 1 -- 172.21.15.129:0/1515337814 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0b4003520 con 0x7fa0c80ccd60 2024-03-23T15:56:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.421+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1515337814 >> v1:172.21.15.157:6789/0 conn(0x7fa0c80ccd60 legacy=0x7fa0c806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.421+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1515337814 >> v1:172.21.15.129:6789/0 conn(0x7fa0c80cef40 legacy=0x7fa0c815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.422+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1515337814 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c80cd950 con 0x7fa0c80ceb00 2024-03-23T15:56:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.426+0000 7fa0cdce2640 1 --2- 172.21.15.129:0/1515337814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c815ad50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80097a0 tx=0x7fa0b802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab240e2eebb47a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:42.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.430+0000 7fa0cd4e1640 1 -- 172.21.15.129:0/1515337814 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b802f810 con 0x7fa0c80ceb00 2024-03-23T15:56:42.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.430+0000 7fa0cd4e1640 1 -- 172.21.15.129:0/1515337814 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b802f970 con 0x7fa0c80ceb00 2024-03-23T15:56:42.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.430+0000 7fa0cd4e1640 1 -- 172.21.15.129:0/1515337814 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b802fd00 con 0x7fa0c80ceb00 2024-03-23T15:56:42.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 -- 172.21.15.129:0/1515337814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 msgr2=0x7fa0c815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1515337814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c815ad50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80097a0 tx=0x7fa0b802b9f0 comp rx=0 tx=0).stop 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 -- 172.21.15.129:0/1515337814 shutdown_connections 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1515337814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c815ad50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 -- 172.21.15.129:0/1515337814 >> 172.21.15.129:0/1515337814 conn(0x7fa0c80b1840 msgr2=0x7fa0c80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 -- 172.21.15.129:0/1515337814 shutdown_connections 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.431+0000 7fa0d0747640 1 -- 172.21.15.129:0/1515337814 wait complete. 2024-03-23T15:56:42.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.432+0000 7fa0d0747640 1 Processor -- start 2024-03-23T15:56:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.432+0000 7fa0d0747640 1 -- start start 2024-03-23T15:56:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0d0747640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0c80ccd60 0x7fa0c8151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0d0747640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c81f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0d0747640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0c80cef40 0x7fa0c81f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:42.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0d0747640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c80c7110 con 0x7fa0c80ceb00 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0d0747640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c80c7ca0 con 0x7fa0c80ccd60 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0d0747640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c8151400 con 0x7fa0c80cef40 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.433+0000 7fa0cdce2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c81f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.434+0000 7fa0cece4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0c80cef40 0x7fa0c81f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.434+0000 7fa0cdce2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c81f3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39808/0 (socket says 172.21.15.129:39808) 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.434+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1916736696 learned_addr learned my addr 172.21.15.129:0/1916736696 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.434+0000 7fa0ce4e3640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0c80ccd60 0x7fa0c8151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:42.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.437+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0c80cef40 msgr2=0x7fa0c81f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:42.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.437+0000 7fa0cdce2640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0c80cef40 0x7fa0c81f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:42.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.437+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0c80ccd60 msgr2=0x7fa0c8151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:42.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.437+0000 7fa0cdce2640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0c80ccd60 0x7fa0c8151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:42.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.437+0000 7fa0cdce2640 1 -- 172.21.15.129:0/1916736696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b8009400 con 0x7fa0c80ceb00 2024-03-23T15:56:42.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.439+0000 7fa0cdce2640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c81f3a90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa0b802c000 tx=0x7fa0b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:42.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.444+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b803c040 con 0x7fa0c80ceb00 2024-03-23T15:56:42.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.444+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b80378f0 con 0x7fa0c80ceb00 2024-03-23T15:56:42.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.444+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c81530b0 con 0x7fa0c80ceb00 2024-03-23T15:56:42.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.444+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8037ca0 con 0x7fa0c80ceb00 2024-03-23T15:56:42.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.444+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c8150ad0 con 0x7fa0c80ceb00 2024-03-23T15:56:42.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.446+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c804f530 con 0x7fa0c80ceb00 2024-03-23T15:56:42.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.468+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa0b802d3a0 con 0x7fa0c80ceb00 2024-03-23T15:56:42.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.469+0000 7fa0beffd640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa0ac06b860 0x7fa0ac06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:42.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.469+0000 7fa0ce4e3640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa0ac06b860 0x7fa0ac06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:42.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.469+0000 7fa0ce4e3640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa0ac06b860 0x7fa0ac06dd20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40011e0 tx=0x7fa0c4007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:42.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.471+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa0b807ac50 con 0x7fa0c80ceb00 2024-03-23T15:56:42.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.731+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa0b8076ef0 con 0x7fa0c80ceb00 2024-03-23T15:56:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:42.994+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0c8075740 con 0x7fa0c80ceb00 2024-03-23T15:56:43.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.018+0000 7fa0beffd640 1 -- 172.21.15.129:0/1916736696 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa0b8047020 con 0x7fa0c80ceb00 2024-03-23T15:56:43.021 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:43.021 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:43.021 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa0ac06b860 msgr2=0x7fa0ac06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa0ac06b860 0x7fa0ac06dd20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40011e0 tx=0x7fa0c4007d30 comp rx=0 tx=0).stop 2024-03-23T15:56:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 msgr2=0x7fa0c81f3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c81f3a90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa0b802c000 tx=0x7fa0b8004730 comp rx=0 tx=0).stop 2024-03-23T15:56:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 shutdown_connections 2024-03-23T15:56:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0c80cef40 0x7fa0c81f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c80ceb00 0x7fa0c81f3a90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa0ac06b860 0x7fa0ac06dd20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 --2- 172.21.15.129:0/1916736696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0c80ccd60 0x7fa0c8151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 >> 172.21.15.129:0/1916736696 conn(0x7fa0c80b1840 msgr2=0x7fa0c80b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 shutdown_connections 2024-03-23T15:56:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:43.021+0000 7fa0d0747640 1 -- 172.21.15.129:0/1916736696 wait complete. 2024-03-23T15:56:43.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:43.643 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:44.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.746+0000 7f79c334f640 1 Processor -- start 2024-03-23T15:56:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.746+0000 7f79c334f640 1 -- start start 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.747+0000 7f79c334f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.747+0000 7f79c334f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f79bc15e040 0x7f79bc0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.747+0000 7f79c334f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79bc06d2c0 con 0x7f79bc06f570 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.747+0000 7f79c334f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f79bc15e740 con 0x7f79bc15e040 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.747+0000 7f79c334f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79bc0ce2e0 con 0x7f79bc15dc60 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.748+0000 7f79c10eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.748+0000 7f79c08ea640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f79bc15e040 0x7f79bc0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:44.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.748+0000 7f79c18ec640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f79bc06f570 0x7f79bc0cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56642/0 (socket says 172.21.15.129:56642) 2024-03-23T15:56:44.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.748+0000 7f79c18ec640 1 -- 172.21.15.129:0/2457435430 learned_addr learned my addr 172.21.15.129:0/2457435430 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.751+0000 7f79c10eb640 1 -- 172.21.15.129:0/2457435430 >> v2:172.21.15.129:3300/0 conn(0x7f79bc15e040 msgr2=0x7f79bc0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.751+0000 7f79c10eb640 1 --2- 172.21.15.129:0/2457435430 >> v2:172.21.15.129:3300/0 conn(0x7f79bc15e040 0x7f79bc0cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.751+0000 7f79c10eb640 1 -- 172.21.15.129:0/2457435430 >> v1:172.21.15.129:6789/0 conn(0x7f79bc06f570 legacy=0x7f79bc0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:44.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.751+0000 7f79c10eb640 1 -- 172.21.15.129:0/2457435430 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc0ccd00 con 0x7f79bc15dc60 2024-03-23T15:56:44.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.753+0000 7f79c10eb640 1 --2- 172.21.15.129:0/2457435430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc06cc50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f79b80097a0 tx=0x7f79b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cc65aa671bb1f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79b3fff640 1 -- 172.21.15.129:0/2457435430 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b802e070 con 0x7f79bc15dc60 2024-03-23T15:56:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79b3fff640 1 -- 172.21.15.129:0/2457435430 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b80315a0 con 0x7f79bc15dc60 2024-03-23T15:56:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79b3fff640 1 -- 172.21.15.129:0/2457435430 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8031930 con 0x7f79bc15dc60 2024-03-23T15:56:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 -- 172.21.15.129:0/2457435430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 msgr2=0x7f79bc06cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 --2- 172.21.15.129:0/2457435430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc06cc50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f79b80097a0 tx=0x7f79b802b840 comp rx=0 tx=0).stop 2024-03-23T15:56:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 -- 172.21.15.129:0/2457435430 shutdown_connections 2024-03-23T15:56:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 --2- 172.21.15.129:0/2457435430 >> v2:172.21.15.129:3300/0 conn(0x7f79bc15e040 0x7f79bc0cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 --2- 172.21.15.129:0/2457435430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc06cc50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 -- 172.21.15.129:0/2457435430 >> 172.21.15.129:0/2457435430 conn(0x7f79bc071300 msgr2=0x7f79bc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.757+0000 7f79c334f640 1 -- 172.21.15.129:0/2457435430 shutdown_connections 2024-03-23T15:56:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.758+0000 7f79c334f640 1 -- 172.21.15.129:0/2457435430 wait complete. 2024-03-23T15:56:44.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 Processor -- start 2024-03-23T15:56:44.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 -- start start 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79bc06f570 0x7f79bc1f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc1f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 0x7f79bc1f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc0b2510 con 0x7f79bc15e040 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc1f3750 con 0x7f79bc06f570 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.759+0000 7f79c334f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79bc1f3a30 con 0x7f79bc15dc60 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.760+0000 7f79c18ec640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 0x7f79bc1f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.760+0000 7f79c08ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc1f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.760+0000 7f79c18ec640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 0x7f79bc1f1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39828/0 (socket says 172.21.15.129:39828) 2024-03-23T15:56:44.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.760+0000 7f79c18ec640 1 -- 172.21.15.129:0/2790540211 learned_addr learned my addr 172.21.15.129:0/2790540211 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:44.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.760+0000 7f79c10eb640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79bc06f570 0x7f79bc1f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:44.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.764+0000 7f79c18ec640 1 -- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 msgr2=0x7f79bc1f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:44.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.764+0000 7f79c18ec640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc1f0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:44.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.764+0000 7f79c18ec640 1 -- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79bc06f570 msgr2=0x7f79bc1f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:44.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.764+0000 7f79c18ec640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79bc06f570 0x7f79bc1f2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:44.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.764+0000 7f79c18ec640 1 -- 172.21.15.129:0/2790540211 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b8009400 con 0x7f79bc15e040 2024-03-23T15:56:44.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.766+0000 7f79c18ec640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 0x7f79bc1f1060 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f79b400caa0 tx=0x7f79b400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:44.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.771+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b400ac00 con 0x7f79bc15e040 2024-03-23T15:56:44.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.771+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b400ad60 con 0x7f79bc15e040 2024-03-23T15:56:44.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.771+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79bc0b1b70 con 0x7f79bc15e040 2024-03-23T15:56:44.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.771+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4012600 con 0x7f79bc15e040 2024-03-23T15:56:44.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.772+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79bc0b2130 con 0x7f79bc15e040 2024-03-23T15:56:44.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.773+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79bc0b2650 con 0x7f79bc15e040 2024-03-23T15:56:44.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.796+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f79b4007070 con 0x7f79bc15e040 2024-03-23T15:56:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.796+0000 7f79b1ffb640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f79a006b890 0x7f79a006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.797+0000 7f79c10eb640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f79a006b890 0x7f79a006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.797+0000 7f79c10eb640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f79a006b890 0x7f79a006dd50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f79b8036000 tx=0x7f79b803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:44.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:44.799+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f79b4058b50 con 0x7f79bc15e040 2024-03-23T15:56:45.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.062+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f79b4054df0 con 0x7f79bc15e040 2024-03-23T15:56:45.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.310+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79bc0b2650 con 0x7f79bc15e040 2024-03-23T15:56:45.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.322+0000 7f79b1ffb640 1 -- 172.21.15.129:0/2790540211 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f79bc0b2650 con 0x7f79bc15e040 2024-03-23T15:56:45.324 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:45.324 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:45.325 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:45.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f79a006b890 msgr2=0x7f79a006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f79a006b890 0x7f79a006dd50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f79b8036000 tx=0x7f79b803d040 comp rx=0 tx=0).stop 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 msgr2=0x7f79bc1f1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 0x7f79bc1f1060 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f79b400caa0 tx=0x7f79b400cf60 comp rx=0 tx=0).stop 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 shutdown_connections 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79bc15e040 0x7f79bc1f1060 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79bc15dc60 0x7f79bc1f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f79a006b890 0x7f79a006dd50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 --2- 172.21.15.129:0/2790540211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79bc06f570 0x7f79bc1f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 >> 172.21.15.129:0/2790540211 conn(0x7f79bc071300 msgr2=0x7f79bc0c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 shutdown_connections 2024-03-23T15:56:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:45.325+0000 7f79c334f640 1 -- 172.21.15.129:0/2790540211 wait complete. 2024-03-23T15:56:45.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:45.947 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:46.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.048+0000 7f6aa8d81640 1 Processor -- start 2024-03-23T15:56:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.048+0000 7f6aa8d81640 1 -- start start 2024-03-23T15:56:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.049+0000 7f6aa8d81640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6aa40c7ca0 0x7f6aa406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.049+0000 7f6aa8d81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 0x7f6aa40ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.049+0000 7f6aa8d81640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa415aa90 con 0x7f6aa41544d0 2024-03-23T15:56:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.049+0000 7f6aa8d81640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa415de50 con 0x7f6aa40c7ca0 2024-03-23T15:56:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.049+0000 7f6aa8d81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa40c4a70 con 0x7f6aa415e270 2024-03-23T15:56:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.049+0000 7f6aa2d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 0x7f6aa40ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:47.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.050+0000 7f6aa2d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 0x7f6aa40ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:53460/0 (socket says 172.21.15.129:53460) 2024-03-23T15:56:47.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.050+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1192861983 learned_addr learned my addr 172.21.15.129:0/1192861983 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:47.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.051+0000 7f6aa1d9b640 1 --2- 172.21.15.129:0/1192861983 >> v2:172.21.15.157:3300/0 conn(0x7f6aa40c7ca0 0x7f6aa406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:47.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.055+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1192861983 >> v2:172.21.15.157:3300/0 conn(0x7f6aa40c7ca0 msgr2=0x7f6aa406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:47.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.055+0000 7f6aa2d9d640 1 --2- 172.21.15.129:0/1192861983 >> v2:172.21.15.157:3300/0 conn(0x7f6aa40c7ca0 0x7f6aa406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.055+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1192861983 >> v1:172.21.15.157:6789/0 conn(0x7f6aa41544d0 legacy=0x7f6aa415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:47.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.055+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1192861983 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa415ad70 con 0x7f6aa415e270 2024-03-23T15:56:47.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.058+0000 7f6aa2d9d640 1 --2- 172.21.15.129:0/1192861983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 0x7f6aa40ccb20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009e50 tx=0x7f6a9c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e93ee3709aaaa28 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.061+0000 7f6aa159a640 1 -- 172.21.15.129:0/1192861983 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c02cb00 con 0x7f6aa415e270 2024-03-23T15:56:47.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.065+0000 7f6aa159a640 1 -- 172.21.15.129:0/1192861983 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9c02cc60 con 0x7f6aa415e270 2024-03-23T15:56:47.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.065+0000 7f6aa159a640 1 -- 172.21.15.129:0/1192861983 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0343f0 con 0x7f6aa415e270 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1192861983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 msgr2=0x7f6aa40ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1192861983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 0x7f6aa40ccb20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009e50 tx=0x7f6a9c02bf60 comp rx=0 tx=0).stop 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1192861983 shutdown_connections 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1192861983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa415e270 0x7f6aa40ccb20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1192861983 >> v2:172.21.15.157:3300/0 conn(0x7f6aa40c7ca0 0x7f6aa406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1192861983 >> 172.21.15.129:0/1192861983 conn(0x7f6aa406e230 msgr2=0x7f6aa406e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1192861983 shutdown_connections 2024-03-23T15:56:47.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.066+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1192861983 wait complete. 2024-03-23T15:56:47.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 Processor -- start 2024-03-23T15:56:47.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 -- start start 2024-03-23T15:56:47.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa40c7ca0 0x7f6aa40cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:47.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6aa41544d0 0x7f6aa41501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:47.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6aa415e270 0x7f6aa41506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:47.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa41f0b40 con 0x7f6aa415e270 2024-03-23T15:56:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa41efce0 con 0x7f6aa41544d0 2024-03-23T15:56:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.068+0000 7f6aa8d81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa41effc0 con 0x7f6aa40c7ca0 2024-03-23T15:56:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.069+0000 7f6aa1d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6aa41544d0 0x7f6aa41501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:47.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.070+0000 7f6aa1d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6aa41544d0 0x7f6aa41501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45854/0 (socket says 172.21.15.129:45854) 2024-03-23T15:56:47.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.070+0000 7f6aa1d9b640 1 -- 172.21.15.129:0/1475885425 learned_addr learned my addr 172.21.15.129:0/1475885425 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:47.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.070+0000 7f6aa259c640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa40c7ca0 0x7f6aa40cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:47.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.070+0000 7f6aa2d9d640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6aa415e270 0x7f6aa41506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.074+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa40c7ca0 msgr2=0x7f6aa40cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.074+0000 7f6aa2d9d640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa40c7ca0 0x7f6aa40cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.074+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6aa41544d0 msgr2=0x7f6aa41501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.074+0000 7f6aa2d9d640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6aa41544d0 0x7f6aa41501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.074+0000 7f6aa2d9d640 1 -- 172.21.15.129:0/1475885425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c009b30 con 0x7f6aa415e270 2024-03-23T15:56:47.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.076+0000 7f6aa2d9d640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6aa415e270 0x7f6aa41506e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c032000 tx=0x7f6a9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:47.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.083+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c009510 con 0x7f6aa415e270 2024-03-23T15:56:47.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.083+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9c03e450 con 0x7f6aa415e270 2024-03-23T15:56:47.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.083+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c03e820 con 0x7f6aa415e270 2024-03-23T15:56:47.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.083+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa41f02a0 con 0x7f6aa415e270 2024-03-23T15:56:47.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.084+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa41f0860 con 0x7f6aa415e270 2024-03-23T15:56:47.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.084+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa406f9d0 con 0x7f6aa415e270 2024-03-23T15:56:47.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.365+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6a9c03e980 con 0x7f6aa415e270 2024-03-23T15:56:47.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.365+0000 7f6a92ffd640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a8406b860 0x7f6a8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:47.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.365+0000 7f6aa259c640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a8406b860 0x7f6a8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:47.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.366+0000 7f6aa259c640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a8406b860 0x7f6a8406dd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a94000f60 tx=0x7f6a94009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:47.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.381+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6a9c07bbd0 con 0x7f6aa415e270 2024-03-23T15:56:47.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.381+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a9c077e70 con 0x7f6aa415e270 2024-03-23T15:56:47.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.673+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6aa406bb30 con 0x7f6aa415e270 2024-03-23T15:56:47.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.694+0000 7f6a92ffd640 1 -- 172.21.15.129:0/1475885425 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6a9c043020 con 0x7f6aa415e270 2024-03-23T15:56:47.697 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:47.697 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:47.697 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:47.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.696+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a8406b860 msgr2=0x7f6a8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:47.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.696+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a8406b860 0x7f6a8406dd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a94000f60 tx=0x7f6a94009040 comp rx=0 tx=0).stop 2024-03-23T15:56:47.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.696+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6aa415e270 msgr2=0x7f6aa41506e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:47.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.696+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6aa415e270 0x7f6aa41506e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c032000 tx=0x7f6a9c004a40 comp rx=0 tx=0).stop 2024-03-23T15:56:47.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 shutdown_connections 2024-03-23T15:56:47.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6aa415e270 0x7f6aa41506e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a8406b860 0x7f6a8406dd20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6aa41544d0 0x7f6aa41501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 --2- 172.21.15.129:0/1475885425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6aa40c7ca0 0x7f6aa40cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:47.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 >> 172.21.15.129:0/1475885425 conn(0x7f6aa406e230 msgr2=0x7f6aa415c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:47.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 shutdown_connections 2024-03-23T15:56:47.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:47.697+0000 7f6aa8d81640 1 -- 172.21.15.129:0/1475885425 wait complete. 2024-03-23T15:56:48.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:48.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:49.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.419+0000 7f41994ce640 1 Processor -- start 2024-03-23T15:56:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.419+0000 7f41994ce640 1 -- start start 2024-03-23T15:56:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.420+0000 7f41994ce640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41941544b0 0x7f419415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.420+0000 7f41994ce640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f41940caa00 0x7f419406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.420+0000 7f41994ce640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41940cbcb0 con 0x7f419406c0b0 2024-03-23T15:56:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.420+0000 7f41994ce640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f41940c5be0 con 0x7f41940caa00 2024-03-23T15:56:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.420+0000 7f41994ce640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f419415b280 con 0x7f41941544b0 2024-03-23T15:56:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.421+0000 7f4192ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41941544b0 0x7f419415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:49.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.421+0000 7f41937fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f419406c0b0 0x7f41940c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:56656/0 (socket says 172.21.15.129:56656) 2024-03-23T15:56:49.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.421+0000 7f41937fe640 1 -- 172.21.15.129:0/3450471537 learned_addr learned my addr 172.21.15.129:0/3450471537 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:49.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.421+0000 7f41927fc640 1 --2- 172.21.15.129:0/3450471537 >> v2:172.21.15.129:3300/0 conn(0x7f41940caa00 0x7f419406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.425+0000 7f4192ffd640 1 -- 172.21.15.129:0/3450471537 >> v2:172.21.15.129:3300/0 conn(0x7f41940caa00 msgr2=0x7f419406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.425+0000 7f4192ffd640 1 --2- 172.21.15.129:0/3450471537 >> v2:172.21.15.129:3300/0 conn(0x7f41940caa00 0x7f419406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.425+0000 7f4192ffd640 1 -- 172.21.15.129:0/3450471537 >> v1:172.21.15.129:6789/0 conn(0x7f419406c0b0 legacy=0x7f41940c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:49.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.425+0000 7f4192ffd640 1 -- 172.21.15.129:0/3450471537 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419415a9e0 con 0x7f41941544b0 2024-03-23T15:56:49.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.427+0000 7f4192ffd640 1 --2- 172.21.15.129:0/3450471537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41941544b0 0x7f419415a2e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f419406cd70 tx=0x7f418402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff19bc21e8b99fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:49.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.432+0000 7f4191ffb640 1 -- 172.21.15.129:0/3450471537 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184030610 con 0x7f41941544b0 2024-03-23T15:56:49.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.432+0000 7f4191ffb640 1 -- 172.21.15.129:0/3450471537 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4184030770 con 0x7f41941544b0 2024-03-23T15:56:49.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.432+0000 7f4191ffb640 1 -- 172.21.15.129:0/3450471537 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184030b00 con 0x7f41941544b0 2024-03-23T15:56:49.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 -- 172.21.15.129:0/3450471537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41941544b0 msgr2=0x7f419415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:49.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 --2- 172.21.15.129:0/3450471537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41941544b0 0x7f419415a2e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f419406cd70 tx=0x7f418402b960 comp rx=0 tx=0).stop 2024-03-23T15:56:49.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 -- 172.21.15.129:0/3450471537 shutdown_connections 2024-03-23T15:56:49.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 --2- 172.21.15.129:0/3450471537 >> v2:172.21.15.129:3300/0 conn(0x7f41940caa00 0x7f419406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 --2- 172.21.15.129:0/3450471537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41941544b0 0x7f419415a2e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 -- 172.21.15.129:0/3450471537 >> 172.21.15.129:0/3450471537 conn(0x7f4194061b50 msgr2=0x7f4194061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:49.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 -- 172.21.15.129:0/3450471537 shutdown_connections 2024-03-23T15:56:49.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.433+0000 7f41994ce640 1 -- 172.21.15.129:0/3450471537 wait complete. 2024-03-23T15:56:49.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.435+0000 7f41994ce640 1 Processor -- start 2024-03-23T15:56:49.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.435+0000 7f41994ce640 1 -- start start 2024-03-23T15:56:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41994ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f419406c0b0 0x7f4194155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41994ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 0x7f4194154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41994ce640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41941544b0 0x7f41941476e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41994ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41941561c0 con 0x7f41940caa00 2024-03-23T15:56:49.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41994ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4194157430 con 0x7f419406c0b0 2024-03-23T15:56:49.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41994ce640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41941576f0 con 0x7f41941544b0 2024-03-23T15:56:49.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.436+0000 7f41927fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 0x7f4194154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:49.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.437+0000 7f4192ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f419406c0b0 0x7f4194155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:49.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.437+0000 7f41937fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41941544b0 0x7f41941476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:49.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.437+0000 7f41927fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 0x7f4194154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39854/0 (socket says 172.21.15.129:39854) 2024-03-23T15:56:49.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.437+0000 7f41927fc640 1 -- 172.21.15.129:0/163292986 learned_addr learned my addr 172.21.15.129:0/163292986 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:49.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.440+0000 7f41927fc640 1 -- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41941544b0 msgr2=0x7f41941476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:49.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.440+0000 7f41927fc640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41941544b0 0x7f41941476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.440+0000 7f41927fc640 1 -- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f419406c0b0 msgr2=0x7f4194155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:49.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.440+0000 7f41927fc640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f419406c0b0 0x7f4194155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.440+0000 7f41927fc640 1 -- 172.21.15.129:0/163292986 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4184009400 con 0x7f41940caa00 2024-03-23T15:56:49.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.443+0000 7f41927fc640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 0x7f4194154f60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f418800e9b0 tx=0x7f418800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:49.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.447+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188009670 con 0x7f41940caa00 2024-03-23T15:56:49.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.447+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418800c9a0 con 0x7f41940caa00 2024-03-23T15:56:49.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.447+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41941579b0 con 0x7f41940caa00 2024-03-23T15:56:49.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.447+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418800cd50 con 0x7f41940caa00 2024-03-23T15:56:49.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.447+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4194156900 con 0x7f41940caa00 2024-03-23T15:56:49.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.448+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4194075740 con 0x7f41940caa00 2024-03-23T15:56:49.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.712+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4188013470 con 0x7f41940caa00 2024-03-23T15:56:49.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.712+0000 7f417b7fe640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f418006b840 0x7f418006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:49.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.712+0000 7f4192ffd640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f418006b840 0x7f418006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:49.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.713+0000 7f4192ffd640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f418006b840 0x7f418006dd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4184036000 tx=0x7f418403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.726+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4188014e40 con 0x7f41940caa00 2024-03-23T15:56:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.726+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4188055680 con 0x7f41940caa00 2024-03-23T15:56:49.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.950+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4194156400 con 0x7f41940caa00 2024-03-23T15:56:49.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.962+0000 7f417b7fe640 1 -- 172.21.15.129:0/163292986 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4188017090 con 0x7f41940caa00 2024-03-23T15:56:49.964 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:49.964 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f418006b840 msgr2=0x7f418006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f418006b840 0x7f418006dd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4184036000 tx=0x7f418403b040 comp rx=0 tx=0).stop 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 msgr2=0x7f4194154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 0x7f4194154f60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f418800e9b0 tx=0x7f418800ee70 comp rx=0 tx=0).stop 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 shutdown_connections 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41941544b0 0x7f41941476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41940caa00 0x7f4194154f60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f418006b840 0x7f418006dd00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 --2- 172.21.15.129:0/163292986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f419406c0b0 0x7f4194155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 >> 172.21.15.129:0/163292986 conn(0x7f4194061b50 msgr2=0x7f4194153650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 shutdown_connections 2024-03-23T15:56:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:49.964+0000 7f41994ce640 1 -- 172.21.15.129:0/163292986 wait complete. 2024-03-23T15:56:50.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:50.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:51.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.687+0000 7f6248a03640 1 Processor -- start 2024-03-23T15:56:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.687+0000 7f6248a03640 1 -- start start 2024-03-23T15:56:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.687+0000 7f6248a03640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6244154430 0x7f62440c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.687+0000 7f6248a03640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 0x7f624415abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.688+0000 7f6248a03640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624415b100 con 0x7f624415dfb0 2024-03-23T15:56:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.688+0000 7f6248a03640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f624415db90 con 0x7f6244154430 2024-03-23T15:56:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.688+0000 7f6248a03640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f62440c4a60 con 0x7f62440ca9a0 2024-03-23T15:56:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.688+0000 7f6242d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6244154430 0x7f62440c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.688+0000 7f624259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 0x7f624415abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.689+0000 7f624259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 0x7f624415abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:41180/0 (socket says 172.21.15.129:41180) 2024-03-23T15:56:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.689+0000 7f624259c640 1 -- 172.21.15.129:0/3547328620 learned_addr learned my addr 172.21.15.129:0/3547328620 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:51.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.691+0000 7f624259c640 1 -- 172.21.15.129:0/3547328620 >> v2:172.21.15.157:3300/0 conn(0x7f6244154430 msgr2=0x7f62440c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:51.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.691+0000 7f624259c640 1 --2- 172.21.15.129:0/3547328620 >> v2:172.21.15.157:3300/0 conn(0x7f6244154430 0x7f62440c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:51.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.691+0000 7f624259c640 1 -- 172.21.15.129:0/3547328620 >> v1:172.21.15.157:6789/0 conn(0x7f624415dfb0 legacy=0x7f62440cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:51.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.691+0000 7f624259c640 1 -- 172.21.15.129:0/3547328620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440c4d00 con 0x7f62440ca9a0 2024-03-23T15:56:51.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.693+0000 7f624259c640 1 --2- 172.21.15.129:0/3547328620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 0x7f624415abc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6238009870 tx=0x7f623802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09c01bf8200d5ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.696+0000 7f6241d9b640 1 -- 172.21.15.129:0/3547328620 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238031640 con 0x7f62440ca9a0 2024-03-23T15:56:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.698+0000 7f6241d9b640 1 -- 172.21.15.129:0/3547328620 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62380317a0 con 0x7f62440ca9a0 2024-03-23T15:56:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.698+0000 7f6241d9b640 1 -- 172.21.15.129:0/3547328620 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238031b30 con 0x7f62440ca9a0 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 -- 172.21.15.129:0/3547328620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 msgr2=0x7f624415abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 --2- 172.21.15.129:0/3547328620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 0x7f624415abc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6238009870 tx=0x7f623802b8f0 comp rx=0 tx=0).stop 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 -- 172.21.15.129:0/3547328620 shutdown_connections 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 --2- 172.21.15.129:0/3547328620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62440ca9a0 0x7f624415abc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 --2- 172.21.15.129:0/3547328620 >> v2:172.21.15.157:3300/0 conn(0x7f6244154430 0x7f62440c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 -- 172.21.15.129:0/3547328620 >> 172.21.15.129:0/3547328620 conn(0x7f624406c370 msgr2=0x7f624406c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 -- 172.21.15.129:0/3547328620 shutdown_connections 2024-03-23T15:56:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.699+0000 7f6248a03640 1 -- 172.21.15.129:0/3547328620 wait complete. 2024-03-23T15:56:51.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 Processor -- start 2024-03-23T15:56:51.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 -- start start 2024-03-23T15:56:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62440ca9a0 0x7f6244146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6244154430 0x7f6244146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624415dfb0 0x7f62440b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6244150490 con 0x7f624415dfb0 2024-03-23T15:56:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6244151180 con 0x7f62440ca9a0 2024-03-23T15:56:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.701+0000 7f6248a03640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f62441516b0 con 0x7f6244154430 2024-03-23T15:56:51.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.702+0000 7f624259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6244154430 0x7f6244146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:51.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.702+0000 7f6242d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62440ca9a0 0x7f6244146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:51.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.702+0000 7f624359e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624415dfb0 0x7f62440b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:51.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.702+0000 7f6242d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62440ca9a0 0x7f6244146330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:51974/0 (socket says 172.21.15.129:51974) 2024-03-23T15:56:51.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.702+0000 7f6242d9d640 1 -- 172.21.15.129:0/3296610840 learned_addr learned my addr 172.21.15.129:0/3296610840 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:51.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.706+0000 7f624359e640 1 -- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6244154430 msgr2=0x7f6244146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:51.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.706+0000 7f624359e640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6244154430 0x7f6244146870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:51.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.706+0000 7f624359e640 1 -- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62440ca9a0 msgr2=0x7f6244146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:51.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.706+0000 7f624359e640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62440ca9a0 0x7f6244146330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:51.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.706+0000 7f624359e640 1 -- 172.21.15.129:0/3296610840 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62380094d0 con 0x7f624415dfb0 2024-03-23T15:56:51.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.708+0000 7f624359e640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624415dfb0 0x7f62440b48a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f623c00c700 tx=0x7f623c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:51.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.712+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c003b90 con 0x7f624415dfb0 2024-03-23T15:56:51.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.712+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f623c003d10 con 0x7f624415dfb0 2024-03-23T15:56:51.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.712+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c005310 con 0x7f624415dfb0 2024-03-23T15:56:51.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.712+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6244154b60 con 0x7f624415dfb0 2024-03-23T15:56:51.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.712+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62440b3dc0 con 0x7f624415dfb0 2024-03-23T15:56:51.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.713+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62440756d0 con 0x7f624415dfb0 2024-03-23T15:56:51.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.736+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f623c005470 con 0x7f624415dfb0 2024-03-23T15:56:51.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.737+0000 7f622b7fe640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f623006b810 0x7f623006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:51.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.737+0000 7f6242d9d640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f623006b810 0x7f623006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:51.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:51.737+0000 7f6242d9d640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f623006b810 0x7f623006dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6234009c00 tx=0x7f6234009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.001+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f623c059190 con 0x7f624415dfb0 2024-03-23T15:56:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.001+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f623c055430 con 0x7f624415dfb0 2024-03-23T15:56:52.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.244+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62440cb4e0 con 0x7f624415dfb0 2024-03-23T15:56:52.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.257+0000 7f622b7fe640 1 -- 172.21.15.129:0/3296610840 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f623c058af0 con 0x7f624415dfb0 2024-03-23T15:56:52.260 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:52.260 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:52.260 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f623006b810 msgr2=0x7f623006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f623006b810 0x7f623006dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6234009c00 tx=0x7f6234009190 comp rx=0 tx=0).stop 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624415dfb0 msgr2=0x7f62440b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624415dfb0 0x7f62440b48a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f623c00c700 tx=0x7f623c00cbc0 comp rx=0 tx=0).stop 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 shutdown_connections 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624415dfb0 0x7f62440b48a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6244154430 0x7f6244146870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:52.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f623006b810 0x7f623006dcd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:52.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 --2- 172.21.15.129:0/3296610840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62440ca9a0 0x7f6244146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:52.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 >> 172.21.15.129:0/3296610840 conn(0x7f624406c370 msgr2=0x7f62440cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:52.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 shutdown_connections 2024-03-23T15:56:52.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:52.259+0000 7f6248a03640 1 -- 172.21.15.129:0/3296610840 wait complete. 2024-03-23T15:56:52.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:52.879 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:53.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:53.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.980+0000 7fabfea6f640 1 Processor -- start 2024-03-23T15:56:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.980+0000 7fabfea6f640 1 -- start start 2024-03-23T15:56:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfea6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf80ca4a0 0x7fabf806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfea6f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fabf815dda0 0x7fabf815e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfea6f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabf80cb810 con 0x7fabf815d9c0 2024-03-23T15:56:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfea6f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fabf815e770 con 0x7fabf815dda0 2024-03-23T15:56:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfea6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fabf80cc3b0 con 0x7fabf80ca4a0 2024-03-23T15:56:53.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfd00c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fabf815dda0 0x7fabf815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:53.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfc80b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fabf815d9c0 0x7fabf80cbca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:41018/0 (socket says 172.21.15.129:41018) 2024-03-23T15:56:53.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabfc80b640 1 -- 172.21.15.129:0/1960910087 learned_addr learned my addr 172.21.15.129:0/1960910087 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:53.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.981+0000 7fabf7fff640 1 --2- 172.21.15.129:0/1960910087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf80ca4a0 0x7fabf806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:53.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.984+0000 7fabfd00c640 1 -- 172.21.15.129:0/1960910087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf80ca4a0 msgr2=0x7fabf806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:53.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.984+0000 7fabfd00c640 1 --2- 172.21.15.129:0/1960910087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf80ca4a0 0x7fabf806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:53.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.984+0000 7fabfd00c640 1 -- 172.21.15.129:0/1960910087 >> v1:172.21.15.157:6789/0 conn(0x7fabf815d9c0 legacy=0x7fabf80cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:53.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.984+0000 7fabfd00c640 1 -- 172.21.15.129:0/1960910087 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf80cac00 con 0x7fabf815dda0 2024-03-23T15:56:53.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.986+0000 7fabfd00c640 1 --2- 172.21.15.129:0/1960910087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815dda0 0x7fabf815e230 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009d90 tx=0x7fabf002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b1daf85e798def server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.989+0000 7fabf77fe640 1 -- 172.21.15.129:0/1960910087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf002cc50 con 0x7fabf815dda0 2024-03-23T15:56:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.989+0000 7fabf77fe640 1 -- 172.21.15.129:0/1960910087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf002cdb0 con 0x7fabf815dda0 2024-03-23T15:56:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.989+0000 7fabf77fe640 1 -- 172.21.15.129:0/1960910087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf0034660 con 0x7fabf815dda0 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 -- 172.21.15.129:0/1960910087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815dda0 msgr2=0x7fabf815e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1960910087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815dda0 0x7fabf815e230 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009d90 tx=0x7fabf002bea0 comp rx=0 tx=0).stop 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 -- 172.21.15.129:0/1960910087 shutdown_connections 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1960910087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815dda0 0x7fabf815e230 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1960910087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf80ca4a0 0x7fabf806c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 -- 172.21.15.129:0/1960910087 >> 172.21.15.129:0/1960910087 conn(0x7fabf80cfb20 msgr2=0x7fabf80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 -- 172.21.15.129:0/1960910087 shutdown_connections 2024-03-23T15:56:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.990+0000 7fabfea6f640 1 -- 172.21.15.129:0/1960910087 wait complete. 2024-03-23T15:56:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.992+0000 7fabfea6f640 1 Processor -- start 2024-03-23T15:56:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.992+0000 7fabfea6f640 1 -- start start 2024-03-23T15:56:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfea6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabf80ca4a0 0x7fabf80b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfea6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 0x7fabf8157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfea6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf815dda0 0x7fabf8153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfea6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabf81f0900 con 0x7fabf815d9c0 2024-03-23T15:56:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfea6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fabf81f4010 con 0x7fabf80ca4a0 2024-03-23T15:56:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfea6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fabf81efff0 con 0x7fabf815dda0 2024-03-23T15:56:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfc80b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabf80ca4a0 0x7fabf80b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabfd00c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf815dda0 0x7fabf8153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.993+0000 7fabf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 0x7fabf8157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.994+0000 7fabf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 0x7fabf8157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50800/0 (socket says 172.21.15.129:50800) 2024-03-23T15:56:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.994+0000 7fabf7fff640 1 -- 172.21.15.129:0/1704862330 learned_addr learned my addr 172.21.15.129:0/1704862330 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:53.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.997+0000 7fabf7fff640 1 -- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf815dda0 msgr2=0x7fabf8153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:53.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.997+0000 7fabf7fff640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf815dda0 0x7fabf8153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:53.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.997+0000 7fabf7fff640 1 -- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabf80ca4a0 msgr2=0x7fabf80b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.997+0000 7fabf7fff640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabf80ca4a0 0x7fabf80b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.997+0000 7fabf7fff640 1 -- 172.21.15.129:0/1704862330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf0009a70 con 0x7fabf815d9c0 2024-03-23T15:56:54.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:53.999+0000 7fabf7fff640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 0x7fabf8157100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fabe400e870 tx=0x7fabe400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:54.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.002+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40094f0 con 0x7fabf815d9c0 2024-03-23T15:56:54.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.002+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf81f42b0 con 0x7fabf815d9c0 2024-03-23T15:56:54.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.002+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf81f0590 con 0x7fabf815d9c0 2024-03-23T15:56:54.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.003+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf81f0ab0 con 0x7fabf815d9c0 2024-03-23T15:56:54.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.214+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe4005990 con 0x7fabf815d9c0 2024-03-23T15:56:54.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.310+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4005d00 con 0x7fabf815d9c0 2024-03-23T15:56:54.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.311+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fabe400c940 con 0x7fabf815d9c0 2024-03-23T15:56:54.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.311+0000 7fabf57fa640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabdc06b840 0x7fabdc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:54.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.311+0000 7fabfc80b640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabdc06b840 0x7fabdc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.312+0000 7fabfc80b640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabdc06b840 0x7fabdc06dd00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fabec007fd0 tx=0x7fabec007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:54.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.325+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fabe4059220 con 0x7fabf815d9c0 2024-03-23T15:56:54.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.326+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fabe40554c0 con 0x7fabf815d9c0 2024-03-23T15:56:54.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.549+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabf81f0720 con 0x7fabf815d9c0 2024-03-23T15:56:54.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.561+0000 7fabf57fa640 1 -- 172.21.15.129:0/1704862330 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fabe4021020 con 0x7fabf815d9c0 2024-03-23T15:56:54.563 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:54.564 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:54.564 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:54.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.564+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabdc06b840 msgr2=0x7fabdc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:54.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.564+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabdc06b840 0x7fabdc06dd00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fabec007fd0 tx=0x7fabec007e80 comp rx=0 tx=0).stop 2024-03-23T15:56:54.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 msgr2=0x7fabf8157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 0x7fabf8157100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fabe400e870 tx=0x7fabe400ed30 comp rx=0 tx=0).stop 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 shutdown_connections 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabf815dda0 0x7fabf8153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabf815d9c0 0x7fabf8157100 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabdc06b840 0x7fabdc06dd00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 --2- 172.21.15.129:0/1704862330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabf80ca4a0 0x7fabf80b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 >> 172.21.15.129:0/1704862330 conn(0x7fabf80cfb20 msgr2=0x7fabf80c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 shutdown_connections 2024-03-23T15:56:54.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:54.565+0000 7fabfea6f640 1 -- 172.21.15.129:0/1704862330 wait complete. 2024-03-23T15:56:55.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:55.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:56.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.284+0000 7f4128ac0640 1 Processor -- start 2024-03-23T15:56:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.284+0000 7f4128ac0640 1 -- start start 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.285+0000 7f4128ac0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f412415e4b0 0x7f412406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.285+0000 7f4128ac0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41240cb810 con 0x7f41240ca4a0 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.285+0000 7f4128ac0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41240cc3b0 con 0x7f412415d9c0 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.285+0000 7f4128ac0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f41240cac00 con 0x7f412415e4b0 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.285+0000 7f411bfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f41240ca4a0 0x7f412415dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44150/0 (socket says 172.21.15.129:44150) 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.286+0000 7f411bfff640 1 -- 172.21.15.129:0/3740924915 learned_addr learned my addr 172.21.15.129:0/3740924915 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:56:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.286+0000 7f4122d9d640 1 --2- 172.21.15.129:0/3740924915 >> v2:172.21.15.157:3300/0 conn(0x7f412415e4b0 0x7f412406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:56.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.289+0000 7f4121d9b640 1 -- 172.21.15.129:0/3740924915 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736795452 0 0) 0x7f41240cb810 con 0x7f41240ca4a0 2024-03-23T15:56:56.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.289+0000 7f4121d9b640 1 -- 172.21.15.129:0/3740924915 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4110003520 con 0x7f41240ca4a0 2024-03-23T15:56:56.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.291+0000 7f4122d9d640 1 -- 172.21.15.129:0/3740924915 >> v1:172.21.15.157:6789/0 conn(0x7f412415d9c0 legacy=0x7f41240cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.291+0000 7f4122d9d640 1 -- 172.21.15.129:0/3740924915 >> v1:172.21.15.129:6789/0 conn(0x7f41240ca4a0 legacy=0x7f412415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.291+0000 7f4122d9d640 1 -- 172.21.15.129:0/3740924915 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41240caee0 con 0x7f412415e4b0 2024-03-23T15:56:56.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.297+0000 7f4122d9d640 1 --2- 172.21.15.129:0/3740924915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415e4b0 0x7f412406c5a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f411c009d90 tx=0x7f411c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=697048636f818ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:56.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.301+0000 7f4121d9b640 1 -- 172.21.15.129:0/3740924915 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c02cd00 con 0x7f412415e4b0 2024-03-23T15:56:56.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.301+0000 7f4121d9b640 1 -- 172.21.15.129:0/3740924915 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c02ce60 con 0x7f412415e4b0 2024-03-23T15:56:56.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.301+0000 7f4121d9b640 1 -- 172.21.15.129:0/3740924915 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c034660 con 0x7f412415e4b0 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 -- 172.21.15.129:0/3740924915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415e4b0 msgr2=0x7f412406c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3740924915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415e4b0 0x7f412406c5a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f411c009d90 tx=0x7f411c02bf50 comp rx=0 tx=0).stop 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 -- 172.21.15.129:0/3740924915 shutdown_connections 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3740924915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415e4b0 0x7f412406c5a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 -- 172.21.15.129:0/3740924915 >> 172.21.15.129:0/3740924915 conn(0x7f41240cfb20 msgr2=0x7f41240cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 -- 172.21.15.129:0/3740924915 shutdown_connections 2024-03-23T15:56:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.302+0000 7f4128ac0640 1 -- 172.21.15.129:0/3740924915 wait complete. 2024-03-23T15:56:56.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 Processor -- start 2024-03-23T15:56:56.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 -- start start 2024-03-23T15:56:56.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 0x7f41241f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:56.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415d9c0 0x7f41240c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:56.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f412415e4b0 0x7f4124155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41240c60c0 con 0x7f41240ca4a0 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41240c56f0 con 0x7f412415d9c0 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.304+0000 7f4128ac0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41240c4870 con 0x7f412415e4b0 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.305+0000 7f412259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 0x7f41241f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.305+0000 7f411bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415d9c0 0x7f41240c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.305+0000 7f4122d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f412415e4b0 0x7f4124155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.305+0000 7f412259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 0x7f41241f0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50812/0 (socket says 172.21.15.129:50812) 2024-03-23T15:56:56.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.305+0000 7f412259c640 1 -- 172.21.15.129:0/3157798149 learned_addr learned my addr 172.21.15.129:0/3157798149 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.309+0000 7f412259c640 1 -- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f412415e4b0 msgr2=0x7f4124155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.309+0000 7f412259c640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f412415e4b0 0x7f4124155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.309+0000 7f412259c640 1 -- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415d9c0 msgr2=0x7f41240c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.309+0000 7f412259c640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415d9c0 0x7f41240c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.309+0000 7f412259c640 1 -- 172.21.15.129:0/3157798149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c009a70 con 0x7f41240ca4a0 2024-03-23T15:56:56.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.311+0000 7f412259c640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 0x7f41241f0780 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f411400c400 tx=0x7f411400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:56.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.315+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41140052d0 con 0x7f41240ca4a0 2024-03-23T15:56:56.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.315+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4114005450 con 0x7f41240ca4a0 2024-03-23T15:56:56.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.315+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41240c4ae0 con 0x7f41240ca4a0 2024-03-23T15:56:56.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.315+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114007260 con 0x7f41240ca4a0 2024-03-23T15:56:56.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.315+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41240c50a0 con 0x7f41240ca4a0 2024-03-23T15:56:56.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.316+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4124075740 con 0x7f41240ca4a0 2024-03-23T15:56:56.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.582+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f41140073c0 con 0x7f41240ca4a0 2024-03-23T15:56:56.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.582+0000 7f411affd640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f410406b860 0x7f410406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:56.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.582+0000 7f411bfff640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f410406b860 0x7f410406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:56.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.583+0000 7f411bfff640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f410406b860 0x7f410406dd20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f41241e4d30 tx=0x7f410c007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:56.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.640+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4114007670 con 0x7f41240ca4a0 2024-03-23T15:56:56.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.640+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4114054520 con 0x7f41240ca4a0 2024-03-23T15:56:56.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.885+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4124154780 con 0x7f41240ca4a0 2024-03-23T15:56:56.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.907+0000 7f411affd640 1 -- 172.21.15.129:0/3157798149 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4114057be0 con 0x7f41240ca4a0 2024-03-23T15:56:56.910 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:56.910 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:56.910 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f410406b860 msgr2=0x7f410406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f410406b860 0x7f410406dd20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f41241e4d30 tx=0x7f410c007d30 comp rx=0 tx=0).stop 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 msgr2=0x7f41241f0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 0x7f41241f0780 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f411400c400 tx=0x7f411400c8c0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 shutdown_connections 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f412415e4b0 0x7f4124155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f410406b860 0x7f410406dd20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f412415d9c0 0x7f41240c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 --2- 172.21.15.129:0/3157798149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240ca4a0 0x7f41241f0780 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:56.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 >> 172.21.15.129:0/3157798149 conn(0x7f41240cfb20 msgr2=0x7f412415c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:56.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 shutdown_connections 2024-03-23T15:56:56.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:56.909+0000 7f4128ac0640 1 -- 172.21.15.129:0/3157798149 wait complete. 2024-03-23T15:56:57.531 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:57.531 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:56:58.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 Processor -- start 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 -- start start 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdddc15dc60 0x7fdddc06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdddc06f570 0x7fdddc0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdddc06d2c0 con 0x7fdddc15e040 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fdddc15e740 con 0x7fdddc15dc60 2024-03-23T15:56:58.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.633+0000 7fdde1b9a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdddc0ccd00 con 0x7fdddc06f570 2024-03-23T15:56:58.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.634+0000 7fdddb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdddc15dc60 0x7fdddc06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:58.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.634+0000 7fdddbfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdddc06f570 0x7fdddc0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:58.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.634+0000 7fdddb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdddc15dc60 0x7fdddc06cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50828/0 (socket says 172.21.15.129:50828) 2024-03-23T15:56:58.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.634+0000 7fdddb7fe640 1 -- 172.21.15.129:0/3834714816 learned_addr learned my addr 172.21.15.129:0/3834714816 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:58.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.637+0000 7fdddb7fe640 1 -- 172.21.15.129:0/3834714816 >> v2:172.21.15.157:3300/0 conn(0x7fdddc06f570 msgr2=0x7fdddc0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:56:58.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.637+0000 7fdddb7fe640 1 --2- 172.21.15.129:0/3834714816 >> v2:172.21.15.157:3300/0 conn(0x7fdddc06f570 0x7fdddc0cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:58.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.637+0000 7fdddb7fe640 1 -- 172.21.15.129:0/3834714816 >> v1:172.21.15.157:6789/0 conn(0x7fdddc15e040 legacy=0x7fdddc0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:58.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.637+0000 7fdddb7fe640 1 -- 172.21.15.129:0/3834714816 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdddc0ccfe0 con 0x7fdddc15dc60 2024-03-23T15:56:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.639+0000 7fdddb7fe640 1 --2- 172.21.15.129:0/3834714816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15dc60 0x7fdddc06cc50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdddc15b410 tx=0x7fddcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da2342bd197e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:58.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.642+0000 7fddda7fc640 1 -- 172.21.15.129:0/3834714816 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc031870 con 0x7fdddc15dc60 2024-03-23T15:56:58.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.642+0000 7fddda7fc640 1 -- 172.21.15.129:0/3834714816 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddcc0319d0 con 0x7fdddc15dc60 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.642+0000 7fddda7fc640 1 -- 172.21.15.129:0/3834714816 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc031d60 con 0x7fdddc15dc60 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 -- 172.21.15.129:0/3834714816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15dc60 msgr2=0x7fdddc06cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/3834714816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15dc60 0x7fdddc06cc50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdddc15b410 tx=0x7fddcc02b910 comp rx=0 tx=0).stop 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 -- 172.21.15.129:0/3834714816 shutdown_connections 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/3834714816 >> v2:172.21.15.157:3300/0 conn(0x7fdddc06f570 0x7fdddc0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/3834714816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15dc60 0x7fdddc06cc50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 -- 172.21.15.129:0/3834714816 >> 172.21.15.129:0/3834714816 conn(0x7fdddc071300 msgr2=0x7fdddc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 -- 172.21.15.129:0/3834714816 shutdown_connections 2024-03-23T15:56:58.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.643+0000 7fdde1b9a640 1 -- 172.21.15.129:0/3834714816 wait complete. 2024-03-23T15:56:58.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.645+0000 7fdde1b9a640 1 Processor -- start 2024-03-23T15:56:58.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.645+0000 7fdde1b9a640 1 -- start start 2024-03-23T15:56:58.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.646+0000 7fdde1b9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdddc06f570 0x7fdddc1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:58.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.646+0000 7fdde1b9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdddc15dc60 0x7fdddc0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.646+0000 7fdde1b9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 0x7fdddc0cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.646+0000 7fdde1b9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1eff60 con 0x7fdddc15e040 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.646+0000 7fdde1b9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdddc1f0240 con 0x7fdddc15dc60 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.646+0000 7fdde1b9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdddc1f0520 con 0x7fdddc06f570 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.647+0000 7fdddbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 0x7fdddc0cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.647+0000 7fdddaffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdddc15dc60 0x7fdddc0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.647+0000 7fdddbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 0x7fdddc0cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50842/0 (socket says 172.21.15.129:50842) 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.647+0000 7fdddbfff640 1 -- 172.21.15.129:0/53483048 learned_addr learned my addr 172.21.15.129:0/53483048 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:56:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.647+0000 7fdddb7fe640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdddc06f570 0x7fdddc1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:58.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.650+0000 7fdddbfff640 1 -- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdddc06f570 msgr2=0x7fdddc1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:58.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.650+0000 7fdddbfff640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdddc06f570 0x7fdddc1560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:58.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.650+0000 7fdddbfff640 1 -- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdddc15dc60 msgr2=0x7fdddc0ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:58.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.650+0000 7fdddbfff640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdddc15dc60 0x7fdddc0ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:58.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.650+0000 7fdddbfff640 1 -- 172.21.15.129:0/53483048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc0094d0 con 0x7fdddc15e040 2024-03-23T15:56:58.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.653+0000 7fdddbfff640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 0x7fdddc0cabe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fddd400efc0 tx=0x7fddd400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:58.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.657+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4009ba0 con 0x7fdddc15e040 2024-03-23T15:56:58.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.657+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd400f040 con 0x7fdddc15e040 2024-03-23T15:56:58.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.657+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd40146c0 con 0x7fdddc15e040 2024-03-23T15:56:58.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.657+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdddc1f0800 con 0x7fdddc15e040 2024-03-23T15:56:58.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.657+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdddc0ca180 con 0x7fdddc15e040 2024-03-23T15:56:58.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.658+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdddc0ca3a0 con 0x7fdddc15e040 2024-03-23T15:56:58.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.916+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fddd4014820 con 0x7fdddc15e040 2024-03-23T15:56:58.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.916+0000 7fddc3fff640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fddc806b810 0x7fddc806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:56:58.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.917+0000 7fdddb7fe640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fddc806b810 0x7fddc806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:56:58.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.917+0000 7fdddb7fe640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fddc806b810 0x7fddc806dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fddcc036000 tx=0x7fddcc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:56:58.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.933+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fddd4059be0 con 0x7fdddc15e040 2024-03-23T15:56:58.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:58.933+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fddd4055e80 con 0x7fdddc15e040 2024-03-23T15:56:59.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.158+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdddc075740 con 0x7fdddc15e040 2024-03-23T15:56:59.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.173+0000 7fddc3fff640 1 -- 172.21.15.129:0/53483048 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fddd405ae60 con 0x7fdddc15e040 2024-03-23T15:56:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:56:59.175 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:56:59.176 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:56:59.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.175+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fddc806b810 msgr2=0x7fddc806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.175+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fddc806b810 0x7fddc806dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fddcc036000 tx=0x7fddcc03c040 comp rx=0 tx=0).stop 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 msgr2=0x7fdddc0cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 0x7fdddc0cabe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fddd400efc0 tx=0x7fddd400c490 comp rx=0 tx=0).stop 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 shutdown_connections 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdddc15e040 0x7fdddc0cabe0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fddc806b810 0x7fddc806dcd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdddc15dc60 0x7fdddc0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 --2- 172.21.15.129:0/53483048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdddc06f570 0x7fdddc1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 >> 172.21.15.129:0/53483048 conn(0x7fdddc071300 msgr2=0x7fdddc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 shutdown_connections 2024-03-23T15:56:59.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:56:59.176+0000 7fdde1b9a640 1 -- 172.21.15.129:0/53483048 wait complete. 2024-03-23T15:56:59.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:56:59.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:00.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:00.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.895+0000 7f31e07f4640 1 Processor -- start 2024-03-23T15:57:00.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.895+0000 7f31e07f4640 1 -- start start 2024-03-23T15:57:00.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.896+0000 7f31e07f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.896+0000 7f31e07f4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f31d815e040 0x7f31d80cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.896+0000 7f31e07f4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f31d806f570 0x7f31d80cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.896+0000 7f31e07f4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f31d806d2c0 con 0x7f31d806f570 2024-03-23T15:57:00.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.896+0000 7f31e07f4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f31d815e740 con 0x7f31d815e040 2024-03-23T15:57:00.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.896+0000 7f31e07f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f31d80ce2e0 con 0x7f31d815dc60 2024-03-23T15:57:00.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.897+0000 7f31ded91640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f31d806f570 0x7f31d80cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.897+0000 7f31de590640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d806cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.897+0000 7f31ded91640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f31d806f570 0x7f31d80cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57274/0 (socket says 172.21.15.129:57274) 2024-03-23T15:57:00.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.897+0000 7f31ded91640 1 -- 172.21.15.129:0/515328476 learned_addr learned my addr 172.21.15.129:0/515328476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:00.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.897+0000 7f31ddd8f640 1 --2- 172.21.15.129:0/515328476 >> v2:172.21.15.157:3300/0 conn(0x7f31d815e040 0x7f31d80cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.900+0000 7f31ded91640 1 -- 172.21.15.129:0/515328476 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 msgr2=0x7f31d806cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:00.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.900+0000 7f31ded91640 1 --2- 172.21.15.129:0/515328476 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d806cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.900+0000 7f31ded91640 1 -- 172.21.15.129:0/515328476 >> v2:172.21.15.157:3300/0 conn(0x7f31d815e040 msgr2=0x7f31d80cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:00.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.900+0000 7f31ded91640 1 --2- 172.21.15.129:0/515328476 >> v2:172.21.15.157:3300/0 conn(0x7f31d815e040 0x7f31d80cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.900+0000 7f31ded91640 1 -- 172.21.15.129:0/515328476 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d80ccd00 con 0x7f31d806f570 2024-03-23T15:57:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.902+0000 7f31ded91640 1 --2- 172.21.15.129:0/515328476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80cb7c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f31d80733f0 tx=0x7f31d002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad7d21d72eaa1df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:00.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.904+0000 7f31dd58e640 1 -- 172.21.15.129:0/515328476 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d002c9e0 con 0x7f31d806f570 2024-03-23T15:57:00.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.906+0000 7f31dd58e640 1 -- 172.21.15.129:0/515328476 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d002cb40 con 0x7f31d806f570 2024-03-23T15:57:00.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.906+0000 7f31dd58e640 1 -- 172.21.15.129:0/515328476 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d00343f0 con 0x7f31d806f570 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 -- 172.21.15.129:0/515328476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 msgr2=0x7f31d80cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 --2- 172.21.15.129:0/515328476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80cb7c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f31d80733f0 tx=0x7f31d002be40 comp rx=0 tx=0).stop 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 -- 172.21.15.129:0/515328476 shutdown_connections 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 --2- 172.21.15.129:0/515328476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80cb7c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 --2- 172.21.15.129:0/515328476 >> v2:172.21.15.157:3300/0 conn(0x7f31d815e040 0x7f31d80cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 --2- 172.21.15.129:0/515328476 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d806cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 -- 172.21.15.129:0/515328476 >> 172.21.15.129:0/515328476 conn(0x7f31d8071300 msgr2=0x7f31d80716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 -- 172.21.15.129:0/515328476 shutdown_connections 2024-03-23T15:57:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.907+0000 7f31e07f4640 1 -- 172.21.15.129:0/515328476 wait complete. 2024-03-23T15:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.909+0000 7f31e07f4640 1 Processor -- start 2024-03-23T15:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.909+0000 7f31e07f4640 1 -- start start 2024-03-23T15:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31e07f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31e07f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d8152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31e07f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31d815e040 0x7f31d81efc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31e07f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f31d815a4b0 con 0x7f31d806f570 2024-03-23T15:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31e07f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f31d8150890 con 0x7f31d815e040 2024-03-23T15:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31e07f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f31d814dfb0 con 0x7f31d815dc60 2024-03-23T15:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31de590640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31ddd8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d8152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31ded91640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31d815e040 0x7f31d81efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31de590640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57288/0 (socket says 172.21.15.129:57288) 2024-03-23T15:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.910+0000 7f31de590640 1 -- 172.21.15.129:0/3362488972 learned_addr learned my addr 172.21.15.129:0/3362488972 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:00.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.914+0000 7f31ddd8f640 1 -- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31d815e040 msgr2=0x7f31d81efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:00.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.914+0000 7f31ddd8f640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31d815e040 0x7f31d81efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.914+0000 7f31ddd8f640 1 -- 172.21.15.129:0/3362488972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 msgr2=0x7f31d80c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:00.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.914+0000 7f31ddd8f640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:00.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.914+0000 7f31ddd8f640 1 -- 172.21.15.129:0/3362488972 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d0009b30 con 0x7f31d815dc60 2024-03-23T15:57:00.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.917+0000 7f31ddd8f640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d8152810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f31c800e870 tx=0x7f31c800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.922+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c800ee60 con 0x7f31d815dc60 2024-03-23T15:57:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.922+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c800c940 con 0x7f31d815dc60 2024-03-23T15:57:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.922+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d814e290 con 0x7f31d815dc60 2024-03-23T15:57:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.922+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c800ccd0 con 0x7f31d815dc60 2024-03-23T15:57:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.922+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d814e850 con 0x7f31d815dc60 2024-03-23T15:57:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.923+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d8075740 con 0x7f31d815dc60 2024-03-23T15:57:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.947+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f31c8013470 con 0x7f31d815dc60 2024-03-23T15:57:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.948+0000 7f31ceffd640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f31bc06b860 0x7f31bc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.948+0000 7f31de590640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f31bc06b860 0x7f31bc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:00.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.949+0000 7f31de590640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f31bc06b860 0x7f31bc06dd20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009ae0 tx=0x7f31d4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:00.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:00.951+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f31c8014e40 con 0x7f31d815dc60 2024-03-23T15:57:01.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.214+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f31c80553c0 con 0x7f31d815dc60 2024-03-23T15:57:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.434+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31d80688e0 con 0x7f31d815dc60 2024-03-23T15:57:01.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.450+0000 7f31ceffd640 1 -- 172.21.15.129:0/3362488972 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f31c8021090 con 0x7f31d815dc60 2024-03-23T15:57:01.453 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:01.453 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:01.454 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:01.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f31bc06b860 msgr2=0x7f31bc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:01.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f31bc06b860 0x7f31bc06dd20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009ae0 tx=0x7f31d4009190 comp rx=0 tx=0).stop 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 msgr2=0x7f31d8152810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d8152810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f31c800e870 tx=0x7f31c800ed30 comp rx=0 tx=0).stop 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 shutdown_connections 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f31bc06b860 0x7f31bc06dd20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f31d815e040 0x7f31d81efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f31d815dc60 0x7f31d8152810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 --2- 172.21.15.129:0/3362488972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31d806f570 0x7f31d80c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 >> 172.21.15.129:0/3362488972 conn(0x7f31d8071300 msgr2=0x7f31d80cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 shutdown_connections 2024-03-23T15:57:01.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:01.453+0000 7f31e07f4640 1 -- 172.21.15.129:0/3362488972 wait complete. 2024-03-23T15:57:02.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:02.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:03.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.172+0000 7fa7b7055640 1 Processor -- start 2024-03-23T15:57:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.172+0000 7fa7b7055640 1 -- start start 2024-03-23T15:57:03.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.173+0000 7fa7b7055640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa7b006cf70 0x7fa7b00cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:03.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.173+0000 7fa7b7055640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b0070640 con 0x7fa7b015df80 2024-03-23T15:57:03.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.173+0000 7fa7b7055640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b00c4c00 con 0x7fa7b015dba0 2024-03-23T15:57:03.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.173+0000 7fa7b7055640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b006c240 con 0x7fa7b006cf70 2024-03-23T15:57:03.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.174+0000 7fa7b55f2640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa7b015df80 0x7fa7b01540a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:33770/0 (socket says 172.21.15.129:33770) 2024-03-23T15:57:03.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.174+0000 7fa7b55f2640 1 -- 172.21.15.129:0/1480260212 learned_addr learned my addr 172.21.15.129:0/1480260212 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:03.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.174+0000 7fa7affff640 1 --2- 172.21.15.129:0/1480260212 >> v2:172.21.15.157:3300/0 conn(0x7fa7b006cf70 0x7fa7b00cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:03.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.177+0000 7fa7af7fe640 1 -- 172.21.15.129:0/1480260212 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24819247 0 0) 0x7fa7b0070640 con 0x7fa7b015df80 2024-03-23T15:57:03.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.177+0000 7fa7af7fe640 1 -- 172.21.15.129:0/1480260212 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a0003520 con 0x7fa7b015df80 2024-03-23T15:57:03.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.178+0000 7fa7affff640 1 -- 172.21.15.129:0/1480260212 >> v1:172.21.15.157:6789/0 conn(0x7fa7b015dba0 legacy=0x7fa7b006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.178+0000 7fa7affff640 1 -- 172.21.15.129:0/1480260212 >> v1:172.21.15.129:6789/0 conn(0x7fa7b015df80 legacy=0x7fa7b01540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.178+0000 7fa7affff640 1 -- 172.21.15.129:0/1480260212 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b00cbb20 con 0x7fa7b006cf70 2024-03-23T15:57:03.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.185+0000 7fa7affff640 1 --2- 172.21.15.129:0/1480260212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b00cca30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0097c0 tx=0x7fa79c02b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7331e71ed5b36a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.194+0000 7fa7af7fe640 1 -- 172.21.15.129:0/1480260212 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c030880 con 0x7fa7b006cf70 2024-03-23T15:57:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.194+0000 7fa7af7fe640 1 -- 172.21.15.129:0/1480260212 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79c0309e0 con 0x7fa7b006cf70 2024-03-23T15:57:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.194+0000 7fa7af7fe640 1 -- 172.21.15.129:0/1480260212 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c030d70 con 0x7fa7b006cf70 2024-03-23T15:57:03.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 -- 172.21.15.129:0/1480260212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 msgr2=0x7fa7b00cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1480260212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b00cca30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0097c0 tx=0x7fa79c02b9c0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 -- 172.21.15.129:0/1480260212 shutdown_connections 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1480260212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b00cca30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 -- 172.21.15.129:0/1480260212 >> 172.21.15.129:0/1480260212 conn(0x7fa7b0071300 msgr2=0x7fa7b00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 -- 172.21.15.129:0/1480260212 shutdown_connections 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.195+0000 7fa7b7055640 1 -- 172.21.15.129:0/1480260212 wait complete. 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.196+0000 7fa7b7055640 1 Processor -- start 2024-03-23T15:57:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.196+0000 7fa7b7055640 1 -- start start 2024-03-23T15:57:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.197+0000 7fa7b7055640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b01f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.197+0000 7fa7b7055640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b015dba0 0x7fa7b00c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.197+0000 7fa7b7055640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 0x7fa7b00c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.197+0000 7fa7b7055640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b00c8cd0 con 0x7fa7b015dba0 2024-03-23T15:57:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.197+0000 7fa7b7055640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01f0e30 con 0x7fa7b006cf70 2024-03-23T15:57:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.197+0000 7fa7b7055640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b00c6790 con 0x7fa7b015df80 2024-03-23T15:57:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.198+0000 7fa7b55f2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 0x7fa7b00c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.199+0000 7fa7b55f2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 0x7fa7b00c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51424/0 (socket says 172.21.15.129:51424) 2024-03-23T15:57:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.199+0000 7fa7b55f2640 1 -- 172.21.15.129:0/1905067658 learned_addr learned my addr 172.21.15.129:0/1905067658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:03.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.200+0000 7fa7b4df1640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b01f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.202+0000 7fa7affff640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b015dba0 0x7fa7b00c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:03.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.204+0000 7fa7b55f2640 1 -- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 msgr2=0x7fa7b01f08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.204+0000 7fa7b55f2640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b01f08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.204+0000 7fa7b55f2640 1 -- 172.21.15.129:0/1905067658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b015dba0 msgr2=0x7fa7b00c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.204+0000 7fa7b55f2640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b015dba0 0x7fa7b00c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.204+0000 7fa7b55f2640 1 -- 172.21.15.129:0/1905067658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c0093a0 con 0x7fa7b015df80 2024-03-23T15:57:03.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.210+0000 7fa7b55f2640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 0x7fa7b00c6c20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800cad0 tx=0x7fa7a800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.218+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a800ac30 con 0x7fa7b015df80 2024-03-23T15:57:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.218+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a800ad90 con 0x7fa7b015df80 2024-03-23T15:57:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.218+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8012620 con 0x7fa7b015df80 2024-03-23T15:57:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.218+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b00c5ba0 con 0x7fa7b015df80 2024-03-23T15:57:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.218+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b00c5e60 con 0x7fa7b015df80 2024-03-23T15:57:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.219+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b006bd50 con 0x7fa7b015df80 2024-03-23T15:57:03.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.497+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa7a8007070 con 0x7fa7b015df80 2024-03-23T15:57:03.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.497+0000 7fa7ad7fa640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa79406b860 0x7fa79406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:03.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.498+0000 7fa7b4df1640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa79406b860 0x7fa79406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.498+0000 7fa7b4df1640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa79406b860 0x7fa79406dd20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000f60 tx=0x7fa7a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:03.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.503+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa7a8058da0 con 0x7fa7b015df80 2024-03-23T15:57:03.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.522+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa7a8055040 con 0x7fa7b015df80 2024-03-23T15:57:03.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.741+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7b0075740 con 0x7fa7b015df80 2024-03-23T15:57:03.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.804+0000 7fa7ad7fa640 1 -- 172.21.15.129:0/1905067658 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa7a8017090 con 0x7fa7b015df80 2024-03-23T15:57:03.807 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:03.807 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:03.808 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:03.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa79406b860 msgr2=0x7fa79406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa79406b860 0x7fa79406dd20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000f60 tx=0x7fa7a4009040 comp rx=0 tx=0).stop 2024-03-23T15:57:03.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 msgr2=0x7fa7b00c6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 0x7fa7b00c6c20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800cad0 tx=0x7fa7a800cf90 comp rx=0 tx=0).stop 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 shutdown_connections 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7b015df80 0x7fa7b00c6c20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b015dba0 0x7fa7b00c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa79406b860 0x7fa79406dd20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 --2- 172.21.15.129:0/1905067658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7b006cf70 0x7fa7b01f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 >> 172.21.15.129:0/1905067658 conn(0x7fa7b0071300 msgr2=0x7fa7b00ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 shutdown_connections 2024-03-23T15:57:03.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:03.807+0000 7fa7b7055640 1 -- 172.21.15.129:0/1905067658 wait complete. 2024-03-23T15:57:04.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:04.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:05.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:05.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.525+0000 7f0a92be9640 1 Processor -- start 2024-03-23T15:57:05.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.526+0000 7f0a92be9640 1 -- start start 2024-03-23T15:57:05.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.526+0000 7f0a92be9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c06c030 0x7f0a8c06c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:05.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.526+0000 7f0a92be9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0a8c0cbc60 0x7f0a8c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:05.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.526+0000 7f0a92be9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a8c06cae0 con 0x7f0a8c0cc080 2024-03-23T15:57:05.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.526+0000 7f0a92be9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8c0cabc0 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.526+0000 7f0a92be9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c0caea0 con 0x7f0a8c06c030 2024-03-23T15:57:05.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.527+0000 7f0a91186640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f0a8c0cc080 0x7f0a8c15de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:33784/0 (socket says 172.21.15.129:33784) 2024-03-23T15:57:05.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.527+0000 7f0a91186640 1 -- 172.21.15.129:0/2115982626 learned_addr learned my addr 172.21.15.129:0/2115982626 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:05.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.527+0000 7f0a90985640 1 --2- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c06c030 0x7f0a8c06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:05.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.527+0000 7f0a8bfff640 1 --2- 172.21.15.129:0/2115982626 >> v2:172.21.15.157:3300/0 conn(0x7f0a8c0cbc60 0x7f0a8c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:05.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.530+0000 7f0a8b7fe640 1 -- 172.21.15.129:0/2115982626 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536123340 0 0) 0x7f0a8c06cae0 con 0x7f0a8c0cc080 2024-03-23T15:57:05.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.530+0000 7f0a8b7fe640 1 -- 172.21.15.129:0/2115982626 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a7c003520 con 0x7f0a8c0cc080 2024-03-23T15:57:05.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.530+0000 7f0a8bfff640 1 -- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c06c030 msgr2=0x7f0a8c06c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:05.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.530+0000 7f0a8bfff640 1 --2- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c06c030 0x7f0a8c06c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:05.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.530+0000 7f0a8bfff640 1 -- 172.21.15.129:0/2115982626 >> v1:172.21.15.129:6789/0 conn(0x7f0a8c0cc080 legacy=0x7f0a8c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:05.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.530+0000 7f0a8bfff640 1 -- 172.21.15.129:0/2115982626 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c0cb180 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.532+0000 7f0a8bfff640 1 --2- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15d900 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0a780097a0 tx=0x7f0a7802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dbe9ceea36a1770 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:05.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.536+0000 7f0a8b7fe640 1 -- 172.21.15.129:0/2115982626 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7802e070 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.536+0000 7f0a8b7fe640 1 -- 172.21.15.129:0/2115982626 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a780315a0 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.536+0000 7f0a8b7fe640 1 -- 172.21.15.129:0/2115982626 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78031930 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 -- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c0cbc60 msgr2=0x7f0a8c15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 --2- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15d900 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0a780097a0 tx=0x7f0a7802b840 comp rx=0 tx=0).stop 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 -- 172.21.15.129:0/2115982626 shutdown_connections 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 --2- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15d900 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 --2- 172.21.15.129:0/2115982626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c06c030 0x7f0a8c06c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 -- 172.21.15.129:0/2115982626 >> 172.21.15.129:0/2115982626 conn(0x7f0a8c0cfb20 msgr2=0x7f0a8c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 -- 172.21.15.129:0/2115982626 shutdown_connections 2024-03-23T15:57:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.537+0000 7f0a92be9640 1 -- 172.21.15.129:0/2115982626 wait complete. 2024-03-23T15:57:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.538+0000 7f0a92be9640 1 Processor -- start 2024-03-23T15:57:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.538+0000 7f0a92be9640 1 -- start start 2024-03-23T15:57:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.539+0000 7f0a92be9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c06c030 0x7f0a8c0b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:05.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.539+0000 7f0a92be9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:05.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.539+0000 7f0a92be9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c0cc080 0x7f0a8c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:05.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.539+0000 7f0a92be9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c15d600 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.539+0000 7f0a92be9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c14ed20 con 0x7f0a8c06c030 2024-03-23T15:57:05.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.539+0000 7f0a92be9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c14ef80 con 0x7f0a8c0cc080 2024-03-23T15:57:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.540+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.540+0000 7f0a91186640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c0cc080 0x7f0a8c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.540+0000 7f0a90985640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c06c030 0x7f0a8c0b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.540+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57298/0 (socket says 172.21.15.129:57298) 2024-03-23T15:57:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.540+0000 7f0a8bfff640 1 -- 172.21.15.129:0/3152217288 learned_addr learned my addr 172.21.15.129:0/3152217288 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.544+0000 7f0a8bfff640 1 -- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c0cc080 msgr2=0x7f0a8c15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.544+0000 7f0a8bfff640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c0cc080 0x7f0a8c15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.544+0000 7f0a8bfff640 1 -- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c06c030 msgr2=0x7f0a8c0b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.544+0000 7f0a8bfff640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c06c030 0x7f0a8c0b4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.544+0000 7f0a8bfff640 1 -- 172.21.15.129:0/3152217288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a78009400 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.546+0000 7f0a8bfff640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15b630 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0a78034040 tx=0x7f0a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:05.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.550+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78032840 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.551+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a78031f00 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.551+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c14f240 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.551+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78032c00 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.551+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c14f800 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.552+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c075740 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.575+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0a7804e430 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.575+0000 7f0a897fa640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0a7006b860 0x7f0a7006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:05.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.575+0000 7f0a90985640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0a7006b860 0x7f0a7006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:05.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.576+0000 7f0a90985640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0a7006b860 0x7f0a7006dd20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009aa0 tx=0x7f0a80009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:05.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.855+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0a7807ac10 con 0x7f0a8c0cbc60 2024-03-23T15:57:05.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:05.856+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0a78076eb0 con 0x7f0a8c0cbc60 2024-03-23T15:57:06.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.085+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a8c155890 con 0x7f0a8c0cbc60 2024-03-23T15:57:06.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.106+0000 7f0a897fa640 1 -- 172.21.15.129:0/3152217288 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0a7807a570 con 0x7f0a8c0cbc60 2024-03-23T15:57:06.108 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:06.109 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:06.109 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:06.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0a7006b860 msgr2=0x7f0a7006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0a7006b860 0x7f0a7006dd20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009aa0 tx=0x7f0a80009190 comp rx=0 tx=0).stop 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 msgr2=0x7f0a8c15b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15b630 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0a78034040 tx=0x7f0a780046c0 comp rx=0 tx=0).stop 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 shutdown_connections 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a8c0cc080 0x7f0a8c15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a8c0cbc60 0x7f0a8c15b630 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0a7006b860 0x7f0a7006dd20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 --2- 172.21.15.129:0/3152217288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a8c06c030 0x7f0a8c0b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 >> 172.21.15.129:0/3152217288 conn(0x7f0a8c0cfb20 msgr2=0x7f0a8c155ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 shutdown_connections 2024-03-23T15:57:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:06.108+0000 7f0a92be9640 1 -- 172.21.15.129:0/3152217288 wait complete. 2024-03-23T15:57:06.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:06.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:07.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:07.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.827+0000 7fd06d1a2640 1 Processor -- start 2024-03-23T15:57:07.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.828+0000 7fd06d1a2640 1 -- start start 2024-03-23T15:57:07.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.829+0000 7fd06d1a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 0x7fd0680c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:07.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.829+0000 7fd06d1a2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd068157fb0 con 0x7fd068152510 2024-03-23T15:57:07.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.829+0000 7fd06d1a2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06815b6e0 con 0x7fd068071610 2024-03-23T15:57:07.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.829+0000 7fd06d1a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd06815b9c0 con 0x7fd068071230 2024-03-23T15:57:07.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.830+0000 7fd067fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 0x7fd0680c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:07.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.830+0000 7fd067fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 0x7fd0680c3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51446/0 (socket says 172.21.15.129:51446) 2024-03-23T15:57:07.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.830+0000 7fd067fff640 1 -- 172.21.15.129:0/3061384191 learned_addr learned my addr 172.21.15.129:0/3061384191 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.833+0000 7fd067fff640 1 -- 172.21.15.129:0/3061384191 >> v1:172.21.15.157:6789/0 conn(0x7fd068071610 legacy=0x7fd068151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.833+0000 7fd067fff640 1 -- 172.21.15.129:0/3061384191 >> v1:172.21.15.129:6789/0 conn(0x7fd068152510 legacy=0x7fd0681584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.833+0000 7fd067fff640 1 -- 172.21.15.129:0/3061384191 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06815bca0 con 0x7fd068071230 2024-03-23T15:57:07.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.836+0000 7fd067fff640 1 --2- 172.21.15.129:0/3061384191 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 0x7fd0680c3070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd0580098a0 tx=0x7fd05802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f6836c8c82b0fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:07.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.839+0000 7fd066ffd640 1 -- 172.21.15.129:0/3061384191 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058030840 con 0x7fd068071230 2024-03-23T15:57:07.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.841+0000 7fd066ffd640 1 -- 172.21.15.129:0/3061384191 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0580309a0 con 0x7fd068071230 2024-03-23T15:57:07.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.841+0000 7fd066ffd640 1 -- 172.21.15.129:0/3061384191 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058030d30 con 0x7fd068071230 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.842+0000 7fd06d1a2640 1 -- 172.21.15.129:0/3061384191 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 msgr2=0x7fd0680c3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.842+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/3061384191 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 0x7fd0680c3070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd0580098a0 tx=0x7fd05802b980 comp rx=0 tx=0).stop 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.842+0000 7fd06d1a2640 1 -- 172.21.15.129:0/3061384191 shutdown_connections 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.842+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/3061384191 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071230 0x7fd0680c3070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.843+0000 7fd06d1a2640 1 -- 172.21.15.129:0/3061384191 >> 172.21.15.129:0/3061384191 conn(0x7fd06806b130 msgr2=0x7fd06806f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.843+0000 7fd06d1a2640 1 -- 172.21.15.129:0/3061384191 shutdown_connections 2024-03-23T15:57:07.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.843+0000 7fd06d1a2640 1 -- 172.21.15.129:0/3061384191 wait complete. 2024-03-23T15:57:07.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.844+0000 7fd06d1a2640 1 Processor -- start 2024-03-23T15:57:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.845+0000 7fd06d1a2640 1 -- start start 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06d1a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd068071230 0x7fd0680cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06d1a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071610 0x7fd0680c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06d1a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 0x7fd0681f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06d1a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681efdd0 con 0x7fd068152510 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06d1a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd0680cc660 con 0x7fd068071230 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06d1a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd0680cc940 con 0x7fd068071610 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071610 0x7fd0680c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.846+0000 7fd06c9a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 0x7fd0681f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.847+0000 7fd067fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd068071230 0x7fd0680cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.847+0000 7fd06c9a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 0x7fd0681f0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57306/0 (socket says 172.21.15.129:57306) 2024-03-23T15:57:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.847+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071610 0x7fd0680c5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:51454/0 (socket says 172.21.15.129:51454) 2024-03-23T15:57:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.847+0000 7fd06c9a1640 1 -- 172.21.15.129:0/4157683552 learned_addr learned my addr 172.21.15.129:0/4157683552 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.852+0000 7fd06c9a1640 1 -- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071610 msgr2=0x7fd0680c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.852+0000 7fd06c9a1640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071610 0x7fd0680c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.852+0000 7fd06c9a1640 1 -- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd068071230 msgr2=0x7fd0680cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.852+0000 7fd06c9a1640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd068071230 0x7fd0680cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.852+0000 7fd06c9a1640 1 -- 172.21.15.129:0/4157683552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0580094d0 con 0x7fd068152510 2024-03-23T15:57:07.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.855+0000 7fd06c9a1640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 0x7fd0681f0290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd06000b9e0 tx=0x7fd06000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:07.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.859+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060003b70 con 0x7fd068152510 2024-03-23T15:57:07.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.859+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd060003cd0 con 0x7fd068152510 2024-03-23T15:57:07.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.859+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0680ccc20 con 0x7fd068152510 2024-03-23T15:57:07.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.859+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06000cb60 con 0x7fd068152510 2024-03-23T15:57:07.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.859+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0680cd1e0 con 0x7fd068152510 2024-03-23T15:57:07.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:07.861+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0680cd370 con 0x7fd068152510 2024-03-23T15:57:08.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.124+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd06000ccc0 con 0x7fd068152510 2024-03-23T15:57:08.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.125+0000 7fd064ff9640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd05406b810 0x7fd05406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:08.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.125+0000 7fd067fff640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd05406b810 0x7fd05406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:08.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.126+0000 7fd067fff640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd05406b810 0x7fd05406dcd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd058036000 tx=0x7fd058008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.148+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd060058f60 con 0x7fd068152510 2024-03-23T15:57:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.148+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd060055200 con 0x7fd068152510 2024-03-23T15:57:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.368+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0680756d0 con 0x7fd068152510 2024-03-23T15:57:08.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.380+0000 7fd064ff9640 1 -- 172.21.15.129:0/4157683552 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd060022070 con 0x7fd068152510 2024-03-23T15:57:08.383 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:08.383 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:08.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd05406b810 msgr2=0x7fd05406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:08.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd05406b810 0x7fd05406dcd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd058036000 tx=0x7fd058008030 comp rx=0 tx=0).stop 2024-03-23T15:57:08.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 msgr2=0x7fd0681f0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:08.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 0x7fd0681f0290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd06000b9e0 tx=0x7fd06000bea0 comp rx=0 tx=0).stop 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 shutdown_connections 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd068152510 0x7fd0681f0290 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd068071610 0x7fd0680c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd05406b810 0x7fd05406dcd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 --2- 172.21.15.129:0/4157683552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd068071230 0x7fd0680cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 >> 172.21.15.129:0/4157683552 conn(0x7fd06806b130 msgr2=0x7fd0681e1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 shutdown_connections 2024-03-23T15:57:08.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:08.383+0000 7fd06d1a2640 1 -- 172.21.15.129:0/4157683552 wait complete. 2024-03-23T15:57:08.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:57:08.518 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:57:08.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:08.999 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:10.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.100+0000 7f047c39a640 1 Processor -- start 2024-03-23T15:57:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.100+0000 7f047c39a640 1 -- start start 2024-03-23T15:57:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.101+0000 7f047c39a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f047415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.101+0000 7f047c39a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f047406c830 con 0x7f047415de40 2024-03-23T15:57:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.101+0000 7f047c39a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04740cc370 con 0x7f047406c030 2024-03-23T15:57:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.101+0000 7f047c39a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f047406cae0 con 0x7f047406c410 2024-03-23T15:57:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.102+0000 7f047a937640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f047415de40 0x7f0474164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:33788/0 (socket says 172.21.15.129:33788) 2024-03-23T15:57:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.102+0000 7f047a937640 1 -- 172.21.15.129:0/1682258430 learned_addr learned my addr 172.21.15.129:0/1682258430 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.102+0000 7f0479935640 1 --2- 172.21.15.129:0/1682258430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f047415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:10.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.105+0000 7f0479935640 1 -- 172.21.15.129:0/1682258430 >> v1:172.21.15.157:6789/0 conn(0x7f047406c030 legacy=0x7f04740cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.105+0000 7f0479935640 1 -- 172.21.15.129:0/1682258430 >> v1:172.21.15.129:6789/0 conn(0x7f047415de40 legacy=0x7f0474164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.105+0000 7f0479935640 1 -- 172.21.15.129:0/1682258430 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04740cabc0 con 0x7f047406c410 2024-03-23T15:57:10.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.107+0000 7f0479935640 1 --2- 172.21.15.129:0/1682258430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f047415d900 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f04640097d0 tx=0x7f046402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d32e07c39955b90 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:10.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.110+0000 7f0479134640 1 -- 172.21.15.129:0/1682258430 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464031890 con 0x7f047406c410 2024-03-23T15:57:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.113+0000 7f0479134640 1 -- 172.21.15.129:0/1682258430 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04640319f0 con 0x7f047406c410 2024-03-23T15:57:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.113+0000 7f0479134640 1 -- 172.21.15.129:0/1682258430 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464031d80 con 0x7f047406c410 2024-03-23T15:57:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.113+0000 7f047c39a640 1 -- 172.21.15.129:0/1682258430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 msgr2=0x7f047415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.113+0000 7f047c39a640 1 --2- 172.21.15.129:0/1682258430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f047415d900 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f04640097d0 tx=0x7f046402b930 comp rx=0 tx=0).stop 2024-03-23T15:57:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.114+0000 7f047c39a640 1 -- 172.21.15.129:0/1682258430 shutdown_connections 2024-03-23T15:57:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.114+0000 7f047c39a640 1 --2- 172.21.15.129:0/1682258430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f047415d900 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.114+0000 7f047c39a640 1 -- 172.21.15.129:0/1682258430 >> 172.21.15.129:0/1682258430 conn(0x7f04740cfb20 msgr2=0x7f04740cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.114+0000 7f047c39a640 1 -- 172.21.15.129:0/1682258430 shutdown_connections 2024-03-23T15:57:10.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.114+0000 7f047c39a640 1 -- 172.21.15.129:0/1682258430 wait complete. 2024-03-23T15:57:10.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.115+0000 7f047c39a640 1 Processor -- start 2024-03-23T15:57:10.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.116+0000 7f047c39a640 1 -- start start 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047c39a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f047406c030 0x7f04740c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047c39a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f04741f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047c39a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 0x7f04741f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047c39a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0474156c60 con 0x7f047415de40 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047c39a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04741f4380 con 0x7f047406c030 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047c39a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04741f3800 con 0x7f047406c410 2024-03-23T15:57:10.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047a937640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 0x7f04741f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:10.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f047a136640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f047406c030 0x7f04740c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:10.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.117+0000 7f0479935640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f04741f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:10.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.118+0000 7f047a937640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 0x7f04741f0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57314/0 (socket says 172.21.15.129:57314) 2024-03-23T15:57:10.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.118+0000 7f047a937640 1 -- 172.21.15.129:0/1594480444 learned_addr learned my addr 172.21.15.129:0/1594480444 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:10.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.121+0000 7f047a937640 1 -- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 msgr2=0x7f04741f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.121+0000 7f047a937640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f04741f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.121+0000 7f047a937640 1 -- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f047406c030 msgr2=0x7f04740c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.121+0000 7f047a937640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f047406c030 0x7f04740c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.121+0000 7f047a937640 1 -- 172.21.15.129:0/1594480444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0464009400 con 0x7f047415de40 2024-03-23T15:57:10.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.124+0000 7f047a937640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 0x7f04741f0b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f046c00cbc0 tx=0x7f046c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:10.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.128+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c00ad40 con 0x7f047415de40 2024-03-23T15:57:10.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.128+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046c012400 con 0x7f047415de40 2024-03-23T15:57:10.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.128+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04741f3ae0 con 0x7f047415de40 2024-03-23T15:57:10.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.128+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c00aea0 con 0x7f047415de40 2024-03-23T15:57:10.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.128+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04741f40a0 con 0x7f047415de40 2024-03-23T15:57:10.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.129+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f047415e880 con 0x7f047415de40 2024-03-23T15:57:10.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.147+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f046c007070 con 0x7f047415de40 2024-03-23T15:57:10.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.148+0000 7f046affd640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045806b810 0x7f045806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:10.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.148+0000 7f047a136640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045806b810 0x7f045806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:10.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.149+0000 7f047a136640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045806b810 0x7f045806dcd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04700016e0 tx=0x7f0470009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:10.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.152+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f046c058da0 con 0x7f047415de40 2024-03-23T15:57:10.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.398+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f046c055040 con 0x7f047415de40 2024-03-23T15:57:10.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.673+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0474075740 con 0x7f047415de40 2024-03-23T15:57:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.689+0000 7f046affd640 1 -- 172.21.15.129:0/1594480444 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f046c017090 con 0x7f047415de40 2024-03-23T15:57:10.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:10.691 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045806b810 msgr2=0x7f045806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045806b810 0x7f045806dcd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04700016e0 tx=0x7f0470009040 comp rx=0 tx=0).stop 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 msgr2=0x7f04741f0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 0x7f04741f0b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f046c00cbc0 tx=0x7f046c00a430 comp rx=0 tx=0).stop 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 shutdown_connections 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f047415de40 0x7f04741f0b30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f047406c410 0x7f04741f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045806b810 0x7f045806dcd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 --2- 172.21.15.129:0/1594480444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f047406c030 0x7f04740c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 >> 172.21.15.129:0/1594480444 conn(0x7f04740cfb20 msgr2=0x7f047415b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 shutdown_connections 2024-03-23T15:57:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:10.692+0000 7f047c39a640 1 -- 172.21.15.129:0/1594480444 wait complete. 2024-03-23T15:57:11.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:11.314 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:12.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:12.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.415+0000 7f329d1a1640 1 Processor -- start 2024-03-23T15:57:12.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.415+0000 7f329d1a1640 1 -- start start 2024-03-23T15:57:12.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.416+0000 7f329d1a1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 0x7f32981568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:12.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.416+0000 7f329d1a1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3298072ad0 0x7f3298161190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:12.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.416+0000 7f329d1a1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3298073560 con 0x7f3298156df0 2024-03-23T15:57:12.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.416+0000 7f329d1a1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f32981510e0 con 0x7f3298072ad0 2024-03-23T15:57:12.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.416+0000 7f329d1a1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3298072fc0 con 0x7f3298150d00 2024-03-23T15:57:12.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.417+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 0x7f32981568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:12.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.417+0000 7f329759e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3298072ad0 0x7f3298161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:12.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.417+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 0x7f32981568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:60402/0 (socket says 172.21.15.129:60402) 2024-03-23T15:57:12.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.417+0000 7f3296d9d640 1 -- 172.21.15.129:0/2764976165 learned_addr learned my addr 172.21.15.129:0/2764976165 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:12.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.420+0000 7f3296d9d640 1 -- 172.21.15.129:0/2764976165 >> v2:172.21.15.157:3300/0 conn(0x7f3298072ad0 msgr2=0x7f3298161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:12.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.420+0000 7f3296d9d640 1 --2- 172.21.15.129:0/2764976165 >> v2:172.21.15.157:3300/0 conn(0x7f3298072ad0 0x7f3298161190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.420+0000 7f3296d9d640 1 -- 172.21.15.129:0/2764976165 >> v1:172.21.15.157:6789/0 conn(0x7f3298156df0 legacy=0x7f329806d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:12.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.420+0000 7f3296d9d640 1 -- 172.21.15.129:0/2764976165 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3298073280 con 0x7f3298150d00 2024-03-23T15:57:12.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.422+0000 7f3296d9d640 1 --2- 172.21.15.129:0/2764976165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 0x7f32981568b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f328800aa10 tx=0x7f328802ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d57b359f42118a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.425+0000 7f3295d9b640 1 -- 172.21.15.129:0/2764976165 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328802dc10 con 0x7f3298150d00 2024-03-23T15:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f3295d9b640 1 -- 172.21.15.129:0/2764976165 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328802dd70 con 0x7f3298150d00 2024-03-23T15:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f3295d9b640 1 -- 172.21.15.129:0/2764976165 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288035660 con 0x7f3298150d00 2024-03-23T15:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 -- 172.21.15.129:0/2764976165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 msgr2=0x7f32981568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 --2- 172.21.15.129:0/2764976165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 0x7f32981568b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f328800aa10 tx=0x7f328802ce60 comp rx=0 tx=0).stop 2024-03-23T15:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 -- 172.21.15.129:0/2764976165 shutdown_connections 2024-03-23T15:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 --2- 172.21.15.129:0/2764976165 >> v2:172.21.15.157:3300/0 conn(0x7f3298072ad0 0x7f3298161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 --2- 172.21.15.129:0/2764976165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298150d00 0x7f32981568b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 -- 172.21.15.129:0/2764976165 >> 172.21.15.129:0/2764976165 conn(0x7f32980c8370 msgr2=0x7f32980c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 -- 172.21.15.129:0/2764976165 shutdown_connections 2024-03-23T15:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.428+0000 7f329d1a1640 1 -- 172.21.15.129:0/2764976165 wait complete. 2024-03-23T15:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 Processor -- start 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 -- start start 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 0x7f32981e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298150d00 0x7f329814cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298156df0 0x7f3298148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32981ee880 con 0x7f3298072ad0 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32981f1b80 con 0x7f3298150d00 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.430+0000 7f329d1a1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32981f0fb0 con 0x7f3298156df0 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.431+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 0x7f32981e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.431+0000 7f329759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298156df0 0x7f3298148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.431+0000 7f3296d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 0x7f32981e33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54216/0 (socket says 172.21.15.129:54216) 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.431+0000 7f3296d9d640 1 -- 172.21.15.129:0/940464367 learned_addr learned my addr 172.21.15.129:0/940464367 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.431+0000 7f329659c640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298150d00 0x7f329814cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.435+0000 7f3296d9d640 1 -- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298156df0 msgr2=0x7f3298148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.435+0000 7f3296d9d640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298156df0 0x7f3298148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.435+0000 7f3296d9d640 1 -- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298150d00 msgr2=0x7f329814cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.435+0000 7f3296d9d640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298150d00 0x7f329814cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.435+0000 7f3296d9d640 1 -- 172.21.15.129:0/940464367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328800a670 con 0x7f3298072ad0 2024-03-23T15:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.438+0000 7f3296d9d640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 0x7f32981e33b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f328802cea0 tx=0x7f32880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:12.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.458+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288044070 con 0x7f3298072ad0 2024-03-23T15:57:12.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.458+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3288040480 con 0x7f3298072ad0 2024-03-23T15:57:12.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.458+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32981f1250 con 0x7f3298072ad0 2024-03-23T15:57:12.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.458+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288040830 con 0x7f3298072ad0 2024-03-23T15:57:12.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.458+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32981f1810 con 0x7f3298072ad0 2024-03-23T15:57:12.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.459+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3298075740 con 0x7f3298072ad0 2024-03-23T15:57:12.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.478+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3288040990 con 0x7f3298072ad0 2024-03-23T15:57:12.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.478+0000 7f327f7fe640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f328406b890 0x7f328406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:12.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.479+0000 7f329659c640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f328406b890 0x7f328406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:12.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.479+0000 7f329659c640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f328406b890 0x7f328406dd50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32981e4100 tx=0x7f328c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.727+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f328807b590 con 0x7f3298072ad0 2024-03-23T15:57:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.727+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3288077830 con 0x7f3298072ad0 2024-03-23T15:57:12.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.950+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32980c43b0 con 0x7f3298072ad0 2024-03-23T15:57:12.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.970+0000 7f327f7fe640 1 -- 172.21.15.129:0/940464367 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f328803a090 con 0x7f3298072ad0 2024-03-23T15:57:12.973 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:12.973 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:12.974 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.972+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f328406b890 msgr2=0x7f328406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.972+0000 7f329d1a1640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f328406b890 0x7f328406dd50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32981e4100 tx=0x7f328c009040 comp rx=0 tx=0).stop 2024-03-23T15:57:12.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 msgr2=0x7f32981e33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 0x7f32981e33b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f328802cea0 tx=0x7f32880046c0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 shutdown_connections 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3298156df0 0x7f3298148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f328406b890 0x7f328406dd50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3298150d00 0x7f329814cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 --2- 172.21.15.129:0/940464367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3298072ad0 0x7f32981e33b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 >> 172.21.15.129:0/940464367 conn(0x7f32980c8370 msgr2=0x7f329814a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 shutdown_connections 2024-03-23T15:57:12.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:12.973+0000 7f329d1a1640 1 -- 172.21.15.129:0/940464367 wait complete. 2024-03-23T15:57:13.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:13.593 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:14.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:14.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.693+0000 7f9cc04d4640 1 Processor -- start 2024-03-23T15:57:14.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.693+0000 7f9cc04d4640 1 -- start start 2024-03-23T15:57:14.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.694+0000 7f9cc04d4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9cb806c030 0x7f9cb806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:14.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.694+0000 7f9cc04d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cbc60 0x7f9cb815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:14.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.694+0000 7f9cc04d4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cb806cae0 con 0x7f9cb80cc080 2024-03-23T15:57:14.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.694+0000 7f9cc04d4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9cb80cabc0 con 0x7f9cb806c030 2024-03-23T15:57:14.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.694+0000 7f9cc04d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb80caea0 con 0x7f9cb80cbc60 2024-03-23T15:57:14.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.695+0000 7f9cbe270640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9cb806c030 0x7f9cb806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:14.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.695+0000 7f9cbda6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cbc60 0x7f9cb815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:14.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.695+0000 7f9cbea71640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9cb80cc080 0x7f9cb815de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:47662/0 (socket says 172.21.15.129:47662) 2024-03-23T15:57:14.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.695+0000 7f9cbea71640 1 -- 172.21.15.129:0/1173359586 learned_addr learned my addr 172.21.15.129:0/1173359586 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:14.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.698+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1173359586 >> v2:172.21.15.129:3300/0 conn(0x7f9cb806c030 msgr2=0x7f9cb806c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:14.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.698+0000 7f9cbda6f640 1 --2- 172.21.15.129:0/1173359586 >> v2:172.21.15.129:3300/0 conn(0x7f9cb806c030 0x7f9cb806c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:14.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.698+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1173359586 >> v1:172.21.15.129:6789/0 conn(0x7f9cb80cc080 legacy=0x7f9cb815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:14.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.698+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1173359586 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb80cb180 con 0x7f9cb80cbc60 2024-03-23T15:57:14.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.700+0000 7f9cbda6f640 1 --2- 172.21.15.129:0/1173359586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cbc60 0x7f9cb815d900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80097a0 tx=0x7f9ca802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc80a3c77101d06d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:14.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cbd26e640 1 -- 172.21.15.129:0/1173359586 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca802e070 con 0x7f9cb80cbc60 2024-03-23T15:57:14.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cbd26e640 1 -- 172.21.15.129:0/1173359586 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca80315a0 con 0x7f9cb80cbc60 2024-03-23T15:57:14.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cbd26e640 1 -- 172.21.15.129:0/1173359586 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8031930 con 0x7f9cb80cbc60 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1173359586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cbc60 msgr2=0x7f9cb815d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1173359586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cbc60 0x7f9cb815d900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80097a0 tx=0x7f9ca802b840 comp rx=0 tx=0).stop 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1173359586 shutdown_connections 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1173359586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cbc60 0x7f9cb815d900 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1173359586 >> v2:172.21.15.129:3300/0 conn(0x7f9cb806c030 0x7f9cb806c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1173359586 >> 172.21.15.129:0/1173359586 conn(0x7f9cb80cfb20 msgr2=0x7f9cb80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1173359586 shutdown_connections 2024-03-23T15:57:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.705+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1173359586 wait complete. 2024-03-23T15:57:14.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 Processor -- start 2024-03-23T15:57:14.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 -- start start 2024-03-23T15:57:14.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cb806c030 0x7f9cb814e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:14.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 0x7f9cb81f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:14.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cc080 0x7f9cb80c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:14.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb815e7b0 con 0x7f9cb80cbc60 2024-03-23T15:57:14.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb80c6680 con 0x7f9cb806c030 2024-03-23T15:57:14.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.707+0000 7f9cc04d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb815df10 con 0x7f9cb80cc080 2024-03-23T15:57:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.708+0000 7f9cbda6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 0x7f9cb81f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.708+0000 7f9cbea71640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cc080 0x7f9cb80c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.708+0000 7f9cbda6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 0x7f9cb81f0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54236/0 (socket says 172.21.15.129:54236) 2024-03-23T15:57:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.708+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1114436975 learned_addr learned my addr 172.21.15.129:0/1114436975 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.708+0000 7f9cbe270640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cb806c030 0x7f9cb814e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:14.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.712+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cc080 msgr2=0x7f9cb80c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:14.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.712+0000 7f9cbda6f640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cc080 0x7f9cb80c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:14.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.712+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cb806c030 msgr2=0x7f9cb814e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:14.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.712+0000 7f9cbda6f640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cb806c030 0x7f9cb814e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:14.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.712+0000 7f9cbda6f640 1 -- 172.21.15.129:0/1114436975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca8009400 con 0x7f9cb80cbc60 2024-03-23T15:57:14.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.715+0000 7f9cbda6f640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 0x7f9cb81f0540 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8034040 tx=0x7f9ca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:14.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.717+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8032840 con 0x7f9cb80cbc60 2024-03-23T15:57:14.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.717+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb815e1f0 con 0x7f9cb80cbc60 2024-03-23T15:57:14.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.717+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb80c4e80 con 0x7f9cb80cbc60 2024-03-23T15:57:14.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.718+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cb80c4ce0 con 0x7f9cb80cbc60 2024-03-23T15:57:14.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:14.926+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca80329a0 con 0x7f9cb80cbc60 2024-03-23T15:57:15.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.014+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8032d10 con 0x7f9cb80cbc60 2024-03-23T15:57:15.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.015+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ca804e430 con 0x7f9cb80cbc60 2024-03-23T15:57:15.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.015+0000 7f9caeffd640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9c9c06b810 0x7f9c9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:15.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.015+0000 7f9cbe270640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9c9c06b810 0x7f9c9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:15.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.016+0000 7f9cbe270640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9c9c06b810 0x7f9c9c06dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009aa0 tx=0x7f9cb4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:15.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.029+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9ca807aaa0 con 0x7f9cb80cbc60 2024-03-23T15:57:15.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.029+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ca8076d40 con 0x7f9cb80cbc60 2024-03-23T15:57:15.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.256+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cb80cd900 con 0x7f9cb80cbc60 2024-03-23T15:57:15.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.276+0000 7f9caeffd640 1 -- 172.21.15.129:0/1114436975 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9cb80cd900 con 0x7f9cb80cbc60 2024-03-23T15:57:15.279 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:15.279 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:15.279 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9c9c06b810 msgr2=0x7f9c9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9c9c06b810 0x7f9c9c06dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009aa0 tx=0x7f9cb4009190 comp rx=0 tx=0).stop 2024-03-23T15:57:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 msgr2=0x7f9cb81f0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 0x7f9cb81f0540 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8034040 tx=0x7f9ca80046c0 comp rx=0 tx=0).stop 2024-03-23T15:57:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 shutdown_connections 2024-03-23T15:57:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cb80cc080 0x7f9cb80c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:15.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cb80cbc60 0x7f9cb81f0540 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:15.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9c9c06b810 0x7f9c9c06dcd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:15.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 --2- 172.21.15.129:0/1114436975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cb806c030 0x7f9cb814e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:15.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 >> 172.21.15.129:0/1114436975 conn(0x7f9cb80cfb20 msgr2=0x7f9cb80ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:15.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 shutdown_connections 2024-03-23T15:57:15.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:15.278+0000 7f9cc04d4640 1 -- 172.21.15.129:0/1114436975 wait complete. 2024-03-23T15:57:15.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:15.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:16.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 Processor -- start 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 -- start start 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f735006c030 0x7f735006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f73500cbc60 0x7f735015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f735006cae0 con 0x7f73500cc080 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f73500cabc0 con 0x7f735006c030 2024-03-23T15:57:17.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.998+0000 7f7356484640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f73500caea0 con 0x7f73500cbc60 2024-03-23T15:57:17.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.999+0000 7f734ffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f735006c030 0x7f735006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:17.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.999+0000 7f734f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f73500cbc60 0x7f735015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:17.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.999+0000 7f734ffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f735006c030 0x7f735006c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54244/0 (socket says 172.21.15.129:54244) 2024-03-23T15:57:17.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:16.999+0000 7f734ffff640 1 -- 172.21.15.129:0/1282676196 learned_addr learned my addr 172.21.15.129:0/1282676196 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.002+0000 7f734ffff640 1 -- 172.21.15.129:0/1282676196 >> v2:172.21.15.157:3300/0 conn(0x7f73500cbc60 msgr2=0x7f735015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.002+0000 7f734ffff640 1 --2- 172.21.15.129:0/1282676196 >> v2:172.21.15.157:3300/0 conn(0x7f73500cbc60 0x7f735015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.002+0000 7f734ffff640 1 -- 172.21.15.129:0/1282676196 >> v1:172.21.15.157:6789/0 conn(0x7f73500cc080 legacy=0x7f735015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.002+0000 7f734ffff640 1 -- 172.21.15.129:0/1282676196 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73500cb180 con 0x7f735006c030 2024-03-23T15:57:17.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.004+0000 7f734ffff640 1 --2- 172.21.15.129:0/1282676196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735006c410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f73500cd660 tx=0x7f734002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed12bfb91f8adbca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.008+0000 7f734effd640 1 -- 172.21.15.129:0/1282676196 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340030860 con 0x7f735006c030 2024-03-23T15:57:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f734effd640 1 -- 172.21.15.129:0/1282676196 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73400309c0 con 0x7f735006c030 2024-03-23T15:57:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f734effd640 1 -- 172.21.15.129:0/1282676196 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340030d50 con 0x7f735006c030 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 -- 172.21.15.129:0/1282676196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 msgr2=0x7f735006c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 --2- 172.21.15.129:0/1282676196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735006c410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f73500cd660 tx=0x7f734002b9a0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 -- 172.21.15.129:0/1282676196 shutdown_connections 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 --2- 172.21.15.129:0/1282676196 >> v2:172.21.15.157:3300/0 conn(0x7f73500cbc60 0x7f735015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 --2- 172.21.15.129:0/1282676196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735006c410 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 -- 172.21.15.129:0/1282676196 >> 172.21.15.129:0/1282676196 conn(0x7f73500cfb20 msgr2=0x7f73500cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 -- 172.21.15.129:0/1282676196 shutdown_connections 2024-03-23T15:57:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.009+0000 7f7356484640 1 -- 172.21.15.129:0/1282676196 wait complete. 2024-03-23T15:57:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.011+0000 7f7356484640 1 Processor -- start 2024-03-23T15:57:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.011+0000 7f7356484640 1 -- start start 2024-03-23T15:57:17.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7356484640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735015db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:17.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7356484640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73500cbc60 0x7f735014de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:17.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7356484640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 0x7f735014e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:17.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7356484640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73501f0030 con 0x7f735006c030 2024-03-23T15:57:17.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7356484640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73501f3430 con 0x7f73500cc080 2024-03-23T15:57:17.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7356484640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73501f36d0 con 0x7f73500cbc60 2024-03-23T15:57:17.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.012+0000 7f7354a21640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 0x7f735014e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:17.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.013+0000 7f734f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73500cbc60 0x7f735014de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:17.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.013+0000 7f734ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735015db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:17.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.013+0000 7f7354a21640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 0x7f735014e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:36524/0 (socket says 172.21.15.129:36524) 2024-03-23T15:57:17.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.013+0000 7f7354a21640 1 -- 172.21.15.129:0/3853501295 learned_addr learned my addr 172.21.15.129:0/3853501295 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.018+0000 7f7354a21640 1 -- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73500cbc60 msgr2=0x7f735014de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.018+0000 7f7354a21640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73500cbc60 0x7f735014de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.018+0000 7f7354a21640 1 -- 172.21.15.129:0/3853501295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 msgr2=0x7f735015db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.018+0000 7f7354a21640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735015db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.018+0000 7f7354a21640 1 -- 172.21.15.129:0/3853501295 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73400094d0 con 0x7f73500cc080 2024-03-23T15:57:17.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.021+0000 7f7354a21640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 0x7f735014e380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f734800c6e0 tx=0x7f734800cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:17.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.026+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348003b70 con 0x7f73500cc080 2024-03-23T15:57:17.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.026+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7348003cf0 con 0x7f73500cc080 2024-03-23T15:57:17.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.026+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348005310 con 0x7f73500cc080 2024-03-23T15:57:17.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.026+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73501f3990 con 0x7f73500cc080 2024-03-23T15:57:17.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.026+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73501f3f50 con 0x7f73500cc080 2024-03-23T15:57:17.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.027+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73501f0170 con 0x7f73500cc080 2024-03-23T15:57:17.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.315+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7348005470 con 0x7f73500cc080 2024-03-23T15:57:17.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.315+0000 7f734cff9640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f733c06b890 0x7f733c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:17.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.316+0000 7f734ffff640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f733c06b890 0x7f733c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:17.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.316+0000 7f734ffff640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f733c06b890 0x7f733c06dd50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7340036000 tx=0x7f7340008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:17.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.322+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7348058f20 con 0x7f73500cc080 2024-03-23T15:57:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.341+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73480551c0 con 0x7f73500cc080 2024-03-23T15:57:17.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.568+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73501f0170 con 0x7f73500cc080 2024-03-23T15:57:17.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.580+0000 7f734cff9640 1 -- 172.21.15.129:0/3853501295 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f73501f0170 con 0x7f73500cc080 2024-03-23T15:57:17.582 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:17.583 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:17.583 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f733c06b890 msgr2=0x7f733c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f733c06b890 0x7f733c06dd50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7340036000 tx=0x7f7340008030 comp rx=0 tx=0).stop 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 msgr2=0x7f735014e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 0x7f735014e380 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f734800c6e0 tx=0x7f734800cba0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 shutdown_connections 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f733c06b890 0x7f733c06dd50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73500cc080 0x7f735014e380 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73500cbc60 0x7f735014de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 --2- 172.21.15.129:0/3853501295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f735006c030 0x7f735015db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 >> 172.21.15.129:0/3853501295 conn(0x7f73500cfb20 msgr2=0x7f73500c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:17.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 shutdown_connections 2024-03-23T15:57:17.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:17.583+0000 7f7356484640 1 -- 172.21.15.129:0/3853501295 wait complete. 2024-03-23T15:57:18.204 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:18.204 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:19.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:19.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.307+0000 7f9e70215640 1 Processor -- start 2024-03-23T15:57:19.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.308+0000 7f9e70215640 1 -- start start 2024-03-23T15:57:19.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.308+0000 7f9e70215640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e6806c030 0x7f9e6806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:19.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.308+0000 7f9e70215640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9e680cbc60 0x7f9e6815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:19.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.308+0000 7f9e70215640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6806cae0 con 0x7f9e680cc080 2024-03-23T15:57:19.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.308+0000 7f9e70215640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9e680cabc0 con 0x7f9e680cbc60 2024-03-23T15:57:19.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.308+0000 7f9e70215640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9e680caea0 con 0x7f9e6806c030 2024-03-23T15:57:19.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.309+0000 7f9e6e7b2640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9e680cc080 0x7f9e6815de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:47672/0 (socket says 172.21.15.129:47672) 2024-03-23T15:57:19.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.309+0000 7f9e6e7b2640 1 -- 172.21.15.129:0/1164522043 learned_addr learned my addr 172.21.15.129:0/1164522043 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:19.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.309+0000 7f9e6d7b0640 1 --2- 172.21.15.129:0/1164522043 >> v2:172.21.15.157:3300/0 conn(0x7f9e680cbc60 0x7f9e6815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:19.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.309+0000 7f9e6dfb1640 1 --2- 172.21.15.129:0/1164522043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e6806c030 0x7f9e6806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:19.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.311+0000 7f9e6cfaf640 1 -- 172.21.15.129:0/1164522043 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2866593476 0 0) 0x7f9e6806cae0 con 0x7f9e680cc080 2024-03-23T15:57:19.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.311+0000 7f9e6cfaf640 1 -- 172.21.15.129:0/1164522043 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e54003520 con 0x7f9e680cc080 2024-03-23T15:57:19.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.313+0000 7f9e6dfb1640 1 -- 172.21.15.129:0/1164522043 >> v2:172.21.15.157:3300/0 conn(0x7f9e680cbc60 msgr2=0x7f9e6815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:19.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.313+0000 7f9e6dfb1640 1 --2- 172.21.15.129:0/1164522043 >> v2:172.21.15.157:3300/0 conn(0x7f9e680cbc60 0x7f9e6815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.313+0000 7f9e6dfb1640 1 -- 172.21.15.129:0/1164522043 >> v1:172.21.15.129:6789/0 conn(0x7f9e680cc080 legacy=0x7f9e6815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:19.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.313+0000 7f9e6dfb1640 1 -- 172.21.15.129:0/1164522043 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e680cb180 con 0x7f9e6806c030 2024-03-23T15:57:19.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.316+0000 7f9e6dfb1640 1 --2- 172.21.15.129:0/1164522043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e6806c030 0x7f9e6806c410 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9e640097a0 tx=0x7f9e6402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb45141afb446e16 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:19.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.321+0000 7f9e6cfaf640 1 -- 172.21.15.129:0/1164522043 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6402e070 con 0x7f9e6806c030 2024-03-23T15:57:19.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.321+0000 7f9e6cfaf640 1 -- 172.21.15.129:0/1164522043 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e640315a0 con 0x7f9e6806c030 2024-03-23T15:57:19.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.321+0000 7f9e6cfaf640 1 -- 172.21.15.129:0/1164522043 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64031930 con 0x7f9e6806c030 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 -- 172.21.15.129:0/1164522043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e6806c030 msgr2=0x7f9e6806c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 --2- 172.21.15.129:0/1164522043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e6806c030 0x7f9e6806c410 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9e640097a0 tx=0x7f9e6402b840 comp rx=0 tx=0).stop 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 -- 172.21.15.129:0/1164522043 shutdown_connections 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 --2- 172.21.15.129:0/1164522043 >> v2:172.21.15.157:3300/0 conn(0x7f9e680cbc60 0x7f9e6815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 --2- 172.21.15.129:0/1164522043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e6806c030 0x7f9e6806c410 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 -- 172.21.15.129:0/1164522043 >> 172.21.15.129:0/1164522043 conn(0x7f9e680cfb20 msgr2=0x7f9e680cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 -- 172.21.15.129:0/1164522043 shutdown_connections 2024-03-23T15:57:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.322+0000 7f9e70215640 1 -- 172.21.15.129:0/1164522043 wait complete. 2024-03-23T15:57:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.323+0000 7f9e70215640 1 Processor -- start 2024-03-23T15:57:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 -- start start 2024-03-23T15:57:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e6806c030 0x7f9e6815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 0x7f9e680c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e680cc080 0x7f9e681f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680c3350 con 0x7f9e680cbc60 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680c3c10 con 0x7f9e6806c030 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.324+0000 7f9e70215640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9e680c4a50 con 0x7f9e680cc080 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.325+0000 7f9e6d7b0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 0x7f9e680c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.325+0000 7f9e6e7b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e680cc080 0x7f9e681f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.325+0000 7f9e6dfb1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e6806c030 0x7f9e6815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:19.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.325+0000 7f9e6d7b0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 0x7f9e680c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54270/0 (socket says 172.21.15.129:54270) 2024-03-23T15:57:19.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.325+0000 7f9e6d7b0640 1 -- 172.21.15.129:0/4231834300 learned_addr learned my addr 172.21.15.129:0/4231834300 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:19.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.328+0000 7f9e6d7b0640 1 -- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e680cc080 msgr2=0x7f9e681f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:19.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.328+0000 7f9e6d7b0640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e680cc080 0x7f9e681f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.328+0000 7f9e6d7b0640 1 -- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e6806c030 msgr2=0x7f9e6815e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:19.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.328+0000 7f9e6d7b0640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e6806c030 0x7f9e6815e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.328+0000 7f9e6d7b0640 1 -- 172.21.15.129:0/4231834300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e64009400 con 0x7f9e680cbc60 2024-03-23T15:57:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.331+0000 7f9e6d7b0640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 0x7f9e680c8d50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800e9b0 tx=0x7f9e5800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:19.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.335+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58009670 con 0x7f9e680cbc60 2024-03-23T15:57:19.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.335+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e5800c9a0 con 0x7f9e680cbc60 2024-03-23T15:57:19.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.335+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e680c3ed0 con 0x7f9e680cbc60 2024-03-23T15:57:19.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.335+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5800cd50 con 0x7f9e680cbc60 2024-03-23T15:57:19.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.335+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e680c4490 con 0x7f9e680cbc60 2024-03-23T15:57:19.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.336+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e68075740 con 0x7f9e680cbc60 2024-03-23T15:57:19.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.359+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9e58013470 con 0x7f9e680cbc60 2024-03-23T15:57:19.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.360+0000 7f9e5e7fc640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9e4c06b890 0x7f9e4c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:19.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.360+0000 7f9e6dfb1640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9e4c06b890 0x7f9e4c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:19.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.360+0000 7f9e6dfb1640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9e4c06b890 0x7f9e4c06dd50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9e64036000 tx=0x7f9e6403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:19.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.630+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9e58014e40 con 0x7f9e680cbc60 2024-03-23T15:57:19.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.630+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9e58055610 con 0x7f9e680cbc60 2024-03-23T15:57:19.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.927+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e680d0a30 con 0x7f9e680cbc60 2024-03-23T15:57:19.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.941+0000 7f9e5e7fc640 1 -- 172.21.15.129:0/4231834300 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9e58017090 con 0x7f9e680cbc60 2024-03-23T15:57:19.944 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:19.944 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:19.944 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9e4c06b890 msgr2=0x7f9e4c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9e4c06b890 0x7f9e4c06dd50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9e64036000 tx=0x7f9e6403d040 comp rx=0 tx=0).stop 2024-03-23T15:57:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 msgr2=0x7f9e680c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 0x7f9e680c8d50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800e9b0 tx=0x7f9e5800ee70 comp rx=0 tx=0).stop 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 shutdown_connections 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e680cc080 0x7f9e681f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e680cbc60 0x7f9e680c8d50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9e4c06b890 0x7f9e4c06dd50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 --2- 172.21.15.129:0/4231834300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e6806c030 0x7f9e6815e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 >> 172.21.15.129:0/4231834300 conn(0x7f9e680cfb20 msgr2=0x7f9e680c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 shutdown_connections 2024-03-23T15:57:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:19.944+0000 7f9e70215640 1 -- 172.21.15.129:0/4231834300 wait complete. 2024-03-23T15:57:20.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:20.562 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:21.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:21.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.662+0000 7f2c9282a640 1 Processor -- start 2024-03-23T15:57:21.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.662+0000 7f2c9282a640 1 -- start start 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c9282a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2c8c0ccd60 0x7f2c8c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c9282a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ceb00 0x7f2c8c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c9282a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2c8c0cef40 0x7f2c8c06c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c9282a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f2c8c15b290 con 0x7f2c8c0ccd60 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c9282a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2c8c0ce800 con 0x7f2c8c0cef40 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c9282a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8c154a70 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.663+0000 7f2c8bfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2c8c0ccd60 0x7f2c8c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.664+0000 7f2c90dc7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2c8c0cef40 0x7f2c8c06c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.664+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ceb00 0x7f2c8c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.664+0000 7f2c8bfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f2c8c0ccd60 0x7f2c8c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36910/0 (socket says 172.21.15.129:36910) 2024-03-23T15:57:21.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.664+0000 7f2c8bfff640 1 -- 172.21.15.129:0/2642197754 learned_addr learned my addr 172.21.15.129:0/2642197754 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.666+0000 7f2c8bfff640 1 -- 172.21.15.129:0/2642197754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ceb00 msgr2=0x7f2c8c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.666+0000 7f2c8bfff640 1 --2- 172.21.15.129:0/2642197754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ceb00 0x7f2c8c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.666+0000 7f2c8bfff640 1 -- 172.21.15.129:0/2642197754 >> v2:172.21.15.157:3300/0 conn(0x7f2c8c0cef40 msgr2=0x7f2c8c06c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.666+0000 7f2c8bfff640 1 --2- 172.21.15.129:0/2642197754 >> v2:172.21.15.157:3300/0 conn(0x7f2c8c0cef40 0x7f2c8c06c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.666+0000 7f2c8bfff640 1 -- 172.21.15.129:0/2642197754 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8c0cd950 con 0x7f2c8c0ccd60 2024-03-23T15:57:21.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.668+0000 7f2c8bfff640 1 --2- 172.21.15.129:0/2642197754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ccd60 0x7f2c8c15ad50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c15e8e0 tx=0x7f2c7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397c74facdd6ffd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:21.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c8affd640 1 -- 172.21.15.129:0/2642197754 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c030860 con 0x7f2c8c0ccd60 2024-03-23T15:57:21.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c8affd640 1 -- 172.21.15.129:0/2642197754 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c7c0309c0 con 0x7f2c8c0ccd60 2024-03-23T15:57:21.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c8affd640 1 -- 172.21.15.129:0/2642197754 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c030d50 con 0x7f2c8c0ccd60 2024-03-23T15:57:21.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 -- 172.21.15.129:0/2642197754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ccd60 msgr2=0x7f2c8c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:21.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 --2- 172.21.15.129:0/2642197754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ccd60 0x7f2c8c15ad50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c15e8e0 tx=0x7f2c7c02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 -- 172.21.15.129:0/2642197754 shutdown_connections 2024-03-23T15:57:21.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 --2- 172.21.15.129:0/2642197754 >> v2:172.21.15.157:3300/0 conn(0x7f2c8c0cef40 0x7f2c8c06c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 --2- 172.21.15.129:0/2642197754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ceb00 0x7f2c8c06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 --2- 172.21.15.129:0/2642197754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ccd60 0x7f2c8c15ad50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 -- 172.21.15.129:0/2642197754 >> 172.21.15.129:0/2642197754 conn(0x7f2c8c0b1840 msgr2=0x7f2c8c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:21.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.672+0000 7f2c9282a640 1 -- 172.21.15.129:0/2642197754 shutdown_connections 2024-03-23T15:57:21.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.673+0000 7f2c9282a640 1 -- 172.21.15.129:0/2642197754 wait complete. 2024-03-23T15:57:21.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.674+0000 7f2c9282a640 1 Processor -- start 2024-03-23T15:57:21.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.675+0000 7f2c9282a640 1 -- start start 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c9282a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ccd60 0x7f2c8c153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c9282a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 0x7f2c8c152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c9282a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c8c0cef40 0x7f2c8c153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c9282a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c0c7890 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c9282a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c0c9ac0 con 0x7f2c8c0cef40 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c9282a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8c0c8ef0 con 0x7f2c8c0ccd60 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 0x7f2c8c152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ccd60 0x7f2c8c153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 0x7f2c8c152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36918/0 (socket says 172.21.15.129:36918) 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.676+0000 7f2c8b7fe640 1 -- 172.21.15.129:0/4085597526 learned_addr learned my addr 172.21.15.129:0/4085597526 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:21.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.677+0000 7f2c90dc7640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c8c0cef40 0x7f2c8c153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.680+0000 7f2c8b7fe640 1 -- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ccd60 msgr2=0x7f2c8c153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:21.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.680+0000 7f2c8b7fe640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ccd60 0x7f2c8c153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.680+0000 7f2c8b7fe640 1 -- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c8c0cef40 msgr2=0x7f2c8c153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:21.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.680+0000 7f2c8b7fe640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c8c0cef40 0x7f2c8c153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:21.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.680+0000 7f2c8b7fe640 1 -- 172.21.15.129:0/4085597526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7c0094d0 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.682+0000 7f2c8b7fe640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 0x7f2c8c152ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000e8a0 tx=0x7f2c8000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.686+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c800094f0 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.686+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c8000c940 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.686+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8000ccf0 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.686+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8c0c9190 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.687+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c8c0c9750 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.688+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c8c04f530 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.706+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2c80013470 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.707+0000 7f2c88ff9640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c7806b810 0x7f2c7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:21.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.707+0000 7f2c8bfff640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c7806b810 0x7f2c7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:21.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.708+0000 7f2c8bfff640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c7806b810 0x7f2c7806dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c036000 tx=0x7f2c7c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:21.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.954+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2c80014e40 con 0x7f2c8c0ceb00 2024-03-23T15:57:21.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:21.954+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c80055480 con 0x7f2c8c0ceb00 2024-03-23T15:57:22.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.176+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c8c06db50 con 0x7f2c8c0ceb00 2024-03-23T15:57:22.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.188+0000 7f2c88ff9640 1 -- 172.21.15.129:0/4085597526 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2c80021090 con 0x7f2c8c0ceb00 2024-03-23T15:57:22.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:22.191 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:22.191 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:22.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c7806b810 msgr2=0x7f2c7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:22.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c7806b810 0x7f2c7806dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c036000 tx=0x7f2c7c008030 comp rx=0 tx=0).stop 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 msgr2=0x7f2c8c152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 0x7f2c8c152ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000e8a0 tx=0x7f2c8000ed60 comp rx=0 tx=0).stop 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 shutdown_connections 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2c7806b810 0x7f2c7806dcd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c8c0cef40 0x7f2c8c153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c8c0ceb00 0x7f2c8c152ed0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 --2- 172.21.15.129:0/4085597526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c8c0ccd60 0x7f2c8c153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 >> 172.21.15.129:0/4085597526 conn(0x7f2c8c0b1840 msgr2=0x7f2c8c0cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 shutdown_connections 2024-03-23T15:57:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:22.191+0000 7f2c9282a640 1 -- 172.21.15.129:0/4085597526 wait complete. 2024-03-23T15:57:22.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:22.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:23.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:23.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.910+0000 7fea53607640 1 Processor -- start 2024-03-23T15:57:23.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.911+0000 7fea53607640 1 -- start start 2024-03-23T15:57:23.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.911+0000 7fea53607640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fea4c1544d0 0x7fea4c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:23.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.911+0000 7fea53607640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea4c15aa90 con 0x7fea4c15e270 2024-03-23T15:57:23.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.911+0000 7fea53607640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea4c15ad70 con 0x7fea4c0c7ca0 2024-03-23T15:57:23.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.911+0000 7fea53607640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fea4c15de50 con 0x7fea4c1544d0 2024-03-23T15:57:23.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.912+0000 7fea51ba4640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fea4c15e270 0x7fea4c0ccb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60322/0 (socket says 172.21.15.129:60322) 2024-03-23T15:57:23.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.912+0000 7fea51ba4640 1 -- 172.21.15.129:0/4232533774 learned_addr learned my addr 172.21.15.129:0/4232533774 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.912+0000 7fea513a3640 1 --2- 172.21.15.129:0/4232533774 >> v2:172.21.15.157:3300/0 conn(0x7fea4c1544d0 0x7fea4c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:23.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.915+0000 7fea43fff640 1 -- 172.21.15.129:0/4232533774 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878923276 0 0) 0x7fea4c15aa90 con 0x7fea4c15e270 2024-03-23T15:57:23.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.915+0000 7fea43fff640 1 -- 172.21.15.129:0/4232533774 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea38003520 con 0x7fea4c15e270 2024-03-23T15:57:23.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.917+0000 7fea513a3640 1 -- 172.21.15.129:0/4232533774 >> v1:172.21.15.157:6789/0 conn(0x7fea4c0c7ca0 legacy=0x7fea4c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:23.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.917+0000 7fea513a3640 1 -- 172.21.15.129:0/4232533774 >> v1:172.21.15.129:6789/0 conn(0x7fea4c15e270 legacy=0x7fea4c0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:23.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.917+0000 7fea513a3640 1 -- 172.21.15.129:0/4232533774 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c15b050 con 0x7fea4c1544d0 2024-03-23T15:57:23.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.921+0000 7fea513a3640 1 --2- 172.21.15.129:0/4232533774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c06bb70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fea480097a0 tx=0x7fea4802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86afc2c4dfb18d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:23.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.925+0000 7fea43fff640 1 -- 172.21.15.129:0/4232533774 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48031830 con 0x7fea4c1544d0 2024-03-23T15:57:23.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.925+0000 7fea43fff640 1 -- 172.21.15.129:0/4232533774 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea48031990 con 0x7fea4c1544d0 2024-03-23T15:57:23.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.925+0000 7fea43fff640 1 -- 172.21.15.129:0/4232533774 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48031d20 con 0x7fea4c1544d0 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 -- 172.21.15.129:0/4232533774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 msgr2=0x7fea4c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 --2- 172.21.15.129:0/4232533774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c06bb70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fea480097a0 tx=0x7fea4802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 -- 172.21.15.129:0/4232533774 shutdown_connections 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 --2- 172.21.15.129:0/4232533774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c06bb70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 -- 172.21.15.129:0/4232533774 >> 172.21.15.129:0/4232533774 conn(0x7fea4c06e230 msgr2=0x7fea4c06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 -- 172.21.15.129:0/4232533774 shutdown_connections 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.926+0000 7fea53607640 1 -- 172.21.15.129:0/4232533774 wait complete. 2024-03-23T15:57:23.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.927+0000 7fea53607640 1 Processor -- start 2024-03-23T15:57:23.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.927+0000 7fea53607640 1 -- start start 2024-03-23T15:57:23.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.928+0000 7fea53607640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea4c0c7ca0 0x7fea4c1f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:23.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.928+0000 7fea53607640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.928+0000 7fea53607640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea4c15e270 0x7fea4c0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.928+0000 7fea53607640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c14d730 con 0x7fea4c0c7ca0 2024-03-23T15:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.928+0000 7fea53607640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c1f0b30 con 0x7fea4c1544d0 2024-03-23T15:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.928+0000 7fea53607640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fea4c15a170 con 0x7fea4c15e270 2024-03-23T15:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.929+0000 7fea513a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea4c0c7ca0 0x7fea4c1f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:23.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.929+0000 7fea50ba2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:23.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.929+0000 7fea51ba4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea4c15e270 0x7fea4c0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:23.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.929+0000 7fea513a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea4c0c7ca0 0x7fea4c1f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36922/0 (socket says 172.21.15.129:36922) 2024-03-23T15:57:23.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.929+0000 7fea513a3640 1 -- 172.21.15.129:0/1536294122 learned_addr learned my addr 172.21.15.129:0/1536294122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.933+0000 7fea51ba4640 1 -- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 msgr2=0x7fea4c158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.933+0000 7fea51ba4640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.933+0000 7fea51ba4640 1 -- 172.21.15.129:0/1536294122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea4c0c7ca0 msgr2=0x7fea4c1f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.933+0000 7fea51ba4640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea4c0c7ca0 0x7fea4c1f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.933+0000 7fea51ba4640 1 -- 172.21.15.129:0/1536294122 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea48009400 con 0x7fea4c15e270 2024-03-23T15:57:23.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.936+0000 7fea51ba4640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea4c15e270 0x7fea4c0c8340 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fea4400caa0 tx=0x7fea4400cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.941+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4400ab40 con 0x7fea4c15e270 2024-03-23T15:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.941+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea4400aca0 con 0x7fea4c15e270 2024-03-23T15:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.941+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44012600 con 0x7fea4c15e270 2024-03-23T15:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.941+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4c155490 con 0x7fea4c15e270 2024-03-23T15:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.941+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4c159e90 con 0x7fea4c15e270 2024-03-23T15:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.942+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4c075740 con 0x7fea4c15e270 2024-03-23T15:57:23.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.967+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea44007070 con 0x7fea4c15e270 2024-03-23T15:57:23.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.967+0000 7fea41ffb640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea3006b890 0x7fea3006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:23.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.967+0000 7fea513a3640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea3006b890 0x7fea3006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:23.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:23.968+0000 7fea513a3640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea3006b890 0x7fea3006dd50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fea4802b910 tx=0x7fea4803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:24.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.250+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fea44058d60 con 0x7fea4c15e270 2024-03-23T15:57:24.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.270+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea44055000 con 0x7fea4c15e270 2024-03-23T15:57:24.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.487+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea4c0c3c70 con 0x7fea4c15e270 2024-03-23T15:57:24.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.508+0000 7fea41ffb640 1 -- 172.21.15.129:0/1536294122 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fea440586c0 con 0x7fea4c15e270 2024-03-23T15:57:24.511 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:24.511 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:24.511 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea3006b890 msgr2=0x7fea3006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea3006b890 0x7fea3006dd50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fea4802b910 tx=0x7fea4803c040 comp rx=0 tx=0).stop 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea4c15e270 msgr2=0x7fea4c0c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea4c15e270 0x7fea4c0c8340 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fea4400caa0 tx=0x7fea4400cea0 comp rx=0 tx=0).stop 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 shutdown_connections 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea4c15e270 0x7fea4c0c8340 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea3006b890 0x7fea3006dd50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea4c1544d0 0x7fea4c158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 --2- 172.21.15.129:0/1536294122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea4c0c7ca0 0x7fea4c1f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:24.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 >> 172.21.15.129:0/1536294122 conn(0x7fea4c06e230 msgr2=0x7fea4c0c4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:24.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 shutdown_connections 2024-03-23T15:57:24.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:24.510+0000 7fea53607640 1 -- 172.21.15.129:0/1536294122 wait complete. 2024-03-23T15:57:25.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:25.129 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:26.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:26.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.231+0000 7f99aa5b5640 1 Processor -- start 2024-03-23T15:57:26.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.231+0000 7f99aa5b5640 1 -- start start 2024-03-23T15:57:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99aa5b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40ccd60 0x7f99a415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99aa5b5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f99a40ceb00 0x7f99a406c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99aa5b5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f99a40cef40 0x7f99a406c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99aa5b5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f99a415b290 con 0x7f99a40ceb00 2024-03-23T15:57:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99aa5b5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f99a40ce800 con 0x7f99a40cef40 2024-03-23T15:57:26.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99aa5b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f99a4154a70 con 0x7f99a40ccd60 2024-03-23T15:57:26.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.233+0000 7f99a37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f99a40ceb00 0x7f99a406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.234+0000 7f99a8b52640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f99a40cef40 0x7f99a406c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.234+0000 7f99a3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40ccd60 0x7f99a415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.234+0000 7f99a37fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f99a40ceb00 0x7f99a406c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36932/0 (socket says 172.21.15.129:36932) 2024-03-23T15:57:26.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.234+0000 7f99a37fe640 1 -- 172.21.15.129:0/3438186845 learned_addr learned my addr 172.21.15.129:0/3438186845 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:26.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.237+0000 7f99a37fe640 1 -- 172.21.15.129:0/3438186845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40ccd60 msgr2=0x7f99a415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:26.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.237+0000 7f99a37fe640 1 --2- 172.21.15.129:0/3438186845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40ccd60 0x7f99a415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.237+0000 7f99a37fe640 1 -- 172.21.15.129:0/3438186845 >> v2:172.21.15.157:3300/0 conn(0x7f99a40cef40 msgr2=0x7f99a406c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:26.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.237+0000 7f99a37fe640 1 --2- 172.21.15.129:0/3438186845 >> v2:172.21.15.157:3300/0 conn(0x7f99a40cef40 0x7f99a406c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.237+0000 7f99a37fe640 1 -- 172.21.15.129:0/3438186845 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a40cd950 con 0x7f99a40ceb00 2024-03-23T15:57:26.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.240+0000 7f99a37fe640 1 --2- 172.21.15.129:0/3438186845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a406c270 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f99a415e8e0 tx=0x7f999802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2e01b1862c07fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:26.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.244+0000 7f99a2ffd640 1 -- 172.21.15.129:0/3438186845 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998030860 con 0x7f99a40ceb00 2024-03-23T15:57:26.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.244+0000 7f99a2ffd640 1 -- 172.21.15.129:0/3438186845 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99980309c0 con 0x7f99a40ceb00 2024-03-23T15:57:26.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.244+0000 7f99a2ffd640 1 -- 172.21.15.129:0/3438186845 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998030d50 con 0x7f99a40ceb00 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 -- 172.21.15.129:0/3438186845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 msgr2=0x7f99a406c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/3438186845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a406c270 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f99a415e8e0 tx=0x7f999802b9a0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 -- 172.21.15.129:0/3438186845 shutdown_connections 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/3438186845 >> v2:172.21.15.157:3300/0 conn(0x7f99a40cef40 0x7f99a406c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/3438186845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a406c270 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/3438186845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40ccd60 0x7f99a415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 -- 172.21.15.129:0/3438186845 >> 172.21.15.129:0/3438186845 conn(0x7f99a40b1840 msgr2=0x7f99a40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 -- 172.21.15.129:0/3438186845 shutdown_connections 2024-03-23T15:57:26.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.245+0000 7f99aa5b5640 1 -- 172.21.15.129:0/3438186845 wait complete. 2024-03-23T15:57:26.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.247+0000 7f99aa5b5640 1 Processor -- start 2024-03-23T15:57:26.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.248+0000 7f99aa5b5640 1 -- start start 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99aa5b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99a40ccd60 0x7f99a4150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99aa5b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a40c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99aa5b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40cef40 0x7f99a40b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99aa5b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99a40c65a0 con 0x7f99a40ceb00 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99aa5b5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f99a40c4aa0 con 0x7f99a40ccd60 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99aa5b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f99a40c40d0 con 0x7f99a40cef40 2024-03-23T15:57:26.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a40c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99a3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99a40ccd60 0x7f99a4150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99a8b52640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40cef40 0x7f99a40b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a40c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36936/0 (socket says 172.21.15.129:36936) 2024-03-23T15:57:26.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.249+0000 7f99a37fe640 1 -- 172.21.15.129:0/1633222482 learned_addr learned my addr 172.21.15.129:0/1633222482 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.253+0000 7f99a37fe640 1 -- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40cef40 msgr2=0x7f99a40b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.253+0000 7f99a37fe640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40cef40 0x7f99a40b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.253+0000 7f99a37fe640 1 -- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99a40ccd60 msgr2=0x7f99a4150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.253+0000 7f99a37fe640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99a40ccd60 0x7f99a4150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.253+0000 7f99a37fe640 1 -- 172.21.15.129:0/1633222482 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99980094d0 con 0x7f99a40ceb00 2024-03-23T15:57:26.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.255+0000 7f99a37fe640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a40c4560 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9998000c00 tx=0x7f9998004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.259+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998039a90 con 0x7f99a40ceb00 2024-03-23T15:57:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.259+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9998039bf0 con 0x7f99a40ceb00 2024-03-23T15:57:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.259+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a40c6050 con 0x7f99a40ceb00 2024-03-23T15:57:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.259+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998043410 con 0x7f99a40ceb00 2024-03-23T15:57:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.259+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a40c37a0 con 0x7f99a40ceb00 2024-03-23T15:57:26.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.260+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a404f530 con 0x7f99a40ceb00 2024-03-23T15:57:26.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.282+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f999804c3b0 con 0x7f99a40ceb00 2024-03-23T15:57:26.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.283+0000 7f99a0ff9640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f999006b860 0x7f999006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:26.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.283+0000 7f99a3fff640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f999006b860 0x7f999006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:26.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.283+0000 7f99a3fff640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f999006b860 0x7f999006dd20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9994009aa0 tx=0x7f9994009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:26.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.286+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f999807b070 con 0x7f99a40ceb00 2024-03-23T15:57:26.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.550+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9998077310 con 0x7f99a40ceb00 2024-03-23T15:57:26.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.772+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99a406db50 con 0x7f99a40ceb00 2024-03-23T15:57:26.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.804+0000 7f99a0ff9640 1 -- 172.21.15.129:0/1633222482 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9998034090 con 0x7f99a40ceb00 2024-03-23T15:57:26.806 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:26.806 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:26.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f999006b860 msgr2=0x7f999006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:26.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f999006b860 0x7f999006dd20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9994009aa0 tx=0x7f9994009190 comp rx=0 tx=0).stop 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 msgr2=0x7f99a40c4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a40c4560 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9998000c00 tx=0x7f9998004730 comp rx=0 tx=0).stop 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 shutdown_connections 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99a40cef40 0x7f99a40b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99a40ceb00 0x7f99a40c4560 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f999006b860 0x7f999006dd20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 --2- 172.21.15.129:0/1633222482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99a40ccd60 0x7f99a4150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 >> 172.21.15.129:0/1633222482 conn(0x7f99a40b1840 msgr2=0x7f99a4158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 shutdown_connections 2024-03-23T15:57:26.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:26.807+0000 7f99aa5b5640 1 -- 172.21.15.129:0/1633222482 wait complete. 2024-03-23T15:57:27.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:27.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:28.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:28.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.529+0000 7f594642a640 1 Processor -- start 2024-03-23T15:57:28.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.529+0000 7f594642a640 1 -- start start 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.530+0000 7f594642a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f59400ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.530+0000 7f594642a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f594015ad10 0x7f594006c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.530+0000 7f594642a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f594015d880 con 0x7f594006c650 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.530+0000 7f594642a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f59400c4ae0 con 0x7f594015ad10 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.530+0000 7f594642a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f59400cd5e0 con 0x7f5940154270 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.531+0000 7f59449c7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f594006c650 0x7f59400c4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43030/0 (socket says 172.21.15.129:43030) 2024-03-23T15:57:28.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.531+0000 7f59449c7640 1 -- 172.21.15.129:0/3261301206 learned_addr learned my addr 172.21.15.129:0/3261301206 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:28.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.531+0000 7f593f7fe640 1 --2- 172.21.15.129:0/3261301206 >> v2:172.21.15.129:3300/0 conn(0x7f594015ad10 0x7f594006c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:28.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.531+0000 7f593ffff640 1 --2- 172.21.15.129:0/3261301206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f59400ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:28.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.535+0000 7f593f7fe640 1 -- 172.21.15.129:0/3261301206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 msgr2=0x7f59400ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:28.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.535+0000 7f593f7fe640 1 --2- 172.21.15.129:0/3261301206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f59400ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:28.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.535+0000 7f593f7fe640 1 -- 172.21.15.129:0/3261301206 >> v1:172.21.15.157:6789/0 conn(0x7f594006c650 legacy=0x7f59400c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:28.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.535+0000 7f593f7fe640 1 -- 172.21.15.129:0/3261301206 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f594015dab0 con 0x7f594015ad10 2024-03-23T15:57:28.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.551+0000 7f593f7fe640 1 --2- 172.21.15.129:0/3261301206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594015ad10 0x7f594006c110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f59340097a0 tx=0x7f593402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d02d1b3744e10f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:28.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.572+0000 7f593effd640 1 -- 172.21.15.129:0/3261301206 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593402f050 con 0x7f594015ad10 2024-03-23T15:57:28.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.572+0000 7f593effd640 1 -- 172.21.15.129:0/3261301206 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5934031700 con 0x7f594015ad10 2024-03-23T15:57:28.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.572+0000 7f593effd640 1 -- 172.21.15.129:0/3261301206 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934031a90 con 0x7f594015ad10 2024-03-23T15:57:28.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 -- 172.21.15.129:0/3261301206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594015ad10 msgr2=0x7f594006c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:28.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 --2- 172.21.15.129:0/3261301206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594015ad10 0x7f594006c110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f59340097a0 tx=0x7f593402b7a0 comp rx=0 tx=0).stop 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 -- 172.21.15.129:0/3261301206 shutdown_connections 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 --2- 172.21.15.129:0/3261301206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594015ad10 0x7f594006c110 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 --2- 172.21.15.129:0/3261301206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f59400ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 -- 172.21.15.129:0/3261301206 >> 172.21.15.129:0/3261301206 conn(0x7f594006d290 msgr2=0x7f5940072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 -- 172.21.15.129:0/3261301206 shutdown_connections 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.573+0000 7f594642a640 1 -- 172.21.15.129:0/3261301206 wait complete. 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.574+0000 7f594642a640 1 Processor -- start 2024-03-23T15:57:28.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.574+0000 7f594642a640 1 -- start start 2024-03-23T15:57:28.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f594642a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 0x7f59401f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:28.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f594642a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f5940144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:28.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f594642a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f594015ad10 0x7f59401413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:28.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f594642a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59401f0840 con 0x7f594006c650 2024-03-23T15:57:28.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f594642a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f594015baf0 con 0x7f594015ad10 2024-03-23T15:57:28.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f594642a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f594015cba0 con 0x7f5940154270 2024-03-23T15:57:28.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.575+0000 7f593ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 0x7f59401f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:28.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.576+0000 7f59449c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f594015ad10 0x7f59401413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:28.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.576+0000 7f593f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f5940144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:28.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.576+0000 7f593ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 0x7f59401f0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36966/0 (socket says 172.21.15.129:36966) 2024-03-23T15:57:28.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.576+0000 7f593ffff640 1 -- 172.21.15.129:0/62612435 learned_addr learned my addr 172.21.15.129:0/62612435 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:28.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.579+0000 7f593ffff640 1 -- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 msgr2=0x7f5940144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:28.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.579+0000 7f593ffff640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f5940144300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:28.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.579+0000 7f593ffff640 1 -- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f594015ad10 msgr2=0x7f59401413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:28.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.579+0000 7f593ffff640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f594015ad10 0x7f59401413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:28.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.579+0000 7f593ffff640 1 -- 172.21.15.129:0/62612435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5934009400 con 0x7f594006c650 2024-03-23T15:57:28.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.581+0000 7f593ffff640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 0x7f59401f0c60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f593000e870 tx=0x7f593000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:28.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.585+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59300094f0 con 0x7f594006c650 2024-03-23T15:57:28.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.585+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f593000c940 con 0x7f594006c650 2024-03-23T15:57:28.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.586+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f594015d670 con 0x7f594006c650 2024-03-23T15:57:28.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.586+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593000ccf0 con 0x7f594006c650 2024-03-23T15:57:28.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.586+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59400ca200 con 0x7f594006c650 2024-03-23T15:57:28.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.587+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f594015e030 con 0x7f594006c650 2024-03-23T15:57:28.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.621+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5930013470 con 0x7f594006c650 2024-03-23T15:57:28.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.622+0000 7f593cff9640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f592c06b810 0x7f592c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:28.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.622+0000 7f593f7fe640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f592c06b810 0x7f592c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:28.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.622+0000 7f593f7fe640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f592c06b810 0x7f592c06dcd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5934035000 tx=0x7f5934008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:28.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.626+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5930014e40 con 0x7f594006c650 2024-03-23T15:57:28.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:28.895+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5930055300 con 0x7f594006c650 2024-03-23T15:57:29.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.152+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f594015e030 con 0x7f594006c650 2024-03-23T15:57:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.173+0000 7f593cff9640 1 -- 172.21.15.129:0/62612435 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f594015e030 con 0x7f594006c650 2024-03-23T15:57:29.176 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:29.176 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:29.177 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:29.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f592c06b810 msgr2=0x7f592c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:29.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f592c06b810 0x7f592c06dcd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5934035000 tx=0x7f5934008030 comp rx=0 tx=0).stop 2024-03-23T15:57:29.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 msgr2=0x7f59401f0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:29.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 0x7f59401f0c60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f593000e870 tx=0x7f593000ed30 comp rx=0 tx=0).stop 2024-03-23T15:57:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 shutdown_connections 2024-03-23T15:57:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f592c06b810 0x7f592c06dcd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f594015ad10 0x7f59401413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5940154270 0x7f5940144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 --2- 172.21.15.129:0/62612435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f594006c650 0x7f59401f0c60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 >> 172.21.15.129:0/62612435 conn(0x7f594006d290 msgr2=0x7f59401469c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:29.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 shutdown_connections 2024-03-23T15:57:29.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:29.175+0000 7f594642a640 1 -- 172.21.15.129:0/62612435 wait complete. 2024-03-23T15:57:29.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:29.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:30.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:30.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.895+0000 7fb2d6cc3640 1 Processor -- start 2024-03-23T15:57:30.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.895+0000 7fb2d6cc3640 1 -- start start 2024-03-23T15:57:30.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.895+0000 7fb2d6cc3640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb2d006c030 0x7fb2d006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:30.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.896+0000 7fb2d6cc3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb2d00cbc60 0x7fb2d015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:30.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.896+0000 7fb2d6cc3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d006cae0 con 0x7fb2d00cc080 2024-03-23T15:57:30.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.896+0000 7fb2d6cc3640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d00cabc0 con 0x7fb2d006c030 2024-03-23T15:57:30.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.896+0000 7fb2d6cc3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d00caea0 con 0x7fb2d00cbc60 2024-03-23T15:57:30.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.896+0000 7fb2d4a5f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb2d006c030 0x7fb2d006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:30.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.896+0000 7fb2cffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb2d00cbc60 0x7fb2d015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:30.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.897+0000 7fb2cffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb2d00cbc60 0x7fb2d015d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38514/0 (socket says 172.21.15.129:38514) 2024-03-23T15:57:30.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.897+0000 7fb2cffff640 1 -- 172.21.15.129:0/1574738456 learned_addr learned my addr 172.21.15.129:0/1574738456 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.899+0000 7fb2cffff640 1 -- 172.21.15.129:0/1574738456 >> v2:172.21.15.129:3300/0 conn(0x7fb2d006c030 msgr2=0x7fb2d006c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.899+0000 7fb2cffff640 1 --2- 172.21.15.129:0/1574738456 >> v2:172.21.15.129:3300/0 conn(0x7fb2d006c030 0x7fb2d006c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.899+0000 7fb2cffff640 1 -- 172.21.15.129:0/1574738456 >> v1:172.21.15.129:6789/0 conn(0x7fb2d00cc080 legacy=0x7fb2d015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.899+0000 7fb2cffff640 1 -- 172.21.15.129:0/1574738456 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d00cb180 con 0x7fb2d00cbc60 2024-03-23T15:57:30.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.901+0000 7fb2cffff640 1 --2- 172.21.15.129:0/1574738456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cbc60 0x7fb2d015d900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00cd660 tx=0x7fb2bc02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b06735ff4e7a1c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:30.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.904+0000 7fb2cf7fe640 1 -- 172.21.15.129:0/1574738456 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc030860 con 0x7fb2d00cbc60 2024-03-23T15:57:30.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.906+0000 7fb2cf7fe640 1 -- 172.21.15.129:0/1574738456 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc0309c0 con 0x7fb2d00cbc60 2024-03-23T15:57:30.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.906+0000 7fb2cf7fe640 1 -- 172.21.15.129:0/1574738456 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc030d50 con 0x7fb2d00cbc60 2024-03-23T15:57:30.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/1574738456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cbc60 msgr2=0x7fb2d015d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:30.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/1574738456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cbc60 0x7fb2d015d900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00cd660 tx=0x7fb2bc02b9a0 comp rx=0 tx=0).stop 2024-03-23T15:57:30.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/1574738456 shutdown_connections 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/1574738456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cbc60 0x7fb2d015d900 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/1574738456 >> v2:172.21.15.129:3300/0 conn(0x7fb2d006c030 0x7fb2d006c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/1574738456 >> 172.21.15.129:0/1574738456 conn(0x7fb2d00cfb20 msgr2=0x7fb2d00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/1574738456 shutdown_connections 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.907+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/1574738456 wait complete. 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.908+0000 7fb2d6cc3640 1 Processor -- start 2024-03-23T15:57:30.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.908+0000 7fb2d6cc3640 1 -- start start 2024-03-23T15:57:30.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d6cc3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d006c030 0x7fb2d014db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:30.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d6cc3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2d00cbc60 0x7fb2d01f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:30.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d6cc3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 0x7fb2d0153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:30.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d6cc3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d01f02a0 con 0x7fb2d006c030 2024-03-23T15:57:30.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d6cc3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d00b2740 con 0x7fb2d00cc080 2024-03-23T15:57:30.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d6cc3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d00b2a00 con 0x7fb2d00cbc60 2024-03-23T15:57:30.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.909+0000 7fb2d5260640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 0x7fb2d0153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:30.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.910+0000 7fb2d5260640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 0x7fb2d0153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38526/0 (socket says 172.21.15.129:38526) 2024-03-23T15:57:30.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.910+0000 7fb2d5260640 1 -- 172.21.15.129:0/326359950 learned_addr learned my addr 172.21.15.129:0/326359950 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:30.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.910+0000 7fb2d4a5f640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d006c030 0x7fb2d014db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:30.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.910+0000 7fb2cffff640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2d00cbc60 0x7fb2d01f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:30.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.913+0000 7fb2d5260640 1 -- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2d00cbc60 msgr2=0x7fb2d01f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:30.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.913+0000 7fb2d5260640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2d00cbc60 0x7fb2d01f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:30.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.913+0000 7fb2d5260640 1 -- 172.21.15.129:0/326359950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d006c030 msgr2=0x7fb2d014db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:30.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.913+0000 7fb2d5260640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d006c030 0x7fb2d014db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:30.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.913+0000 7fb2d5260640 1 -- 172.21.15.129:0/326359950 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2bc0094d0 con 0x7fb2d00cc080 2024-03-23T15:57:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.915+0000 7fb2d5260640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 0x7fb2d0153620 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c6b0 tx=0x7fb2c800cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:30.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.921+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8003b70 con 0x7fb2d00cc080 2024-03-23T15:57:30.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.921+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c8003cf0 con 0x7fb2d00cc080 2024-03-23T15:57:30.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.921+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d00b2ce0 con 0x7fb2d00cc080 2024-03-23T15:57:30.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.921+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8003ea0 con 0x7fb2d00cc080 2024-03-23T15:57:30.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.921+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d01efec0 con 0x7fb2d00cc080 2024-03-23T15:57:30.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:30.922+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2d01f03e0 con 0x7fb2d00cc080 2024-03-23T15:57:31.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.210+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb2c8005310 con 0x7fb2d00cc080 2024-03-23T15:57:31.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.210+0000 7fb2cd7fa640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb2b406b860 0x7fb2b406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:31.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.211+0000 7fb2d4a5f640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb2b406b860 0x7fb2b406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:31.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.211+0000 7fb2d4a5f640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb2b406b860 0x7fb2b406dd20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009b40 tx=0x7fb2c4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:31.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.215+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb2c8058f90 con 0x7fb2d00cc080 2024-03-23T15:57:31.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.226+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb2c8055230 con 0x7fb2d00cc080 2024-03-23T15:57:31.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.454+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2d01f03e0 con 0x7fb2d00cc080 2024-03-23T15:57:31.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.537+0000 7fb2cd7fa640 1 -- 172.21.15.129:0/326359950 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb2d01f03e0 con 0x7fb2d00cc080 2024-03-23T15:57:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:31.540 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:31.540 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb2b406b860 msgr2=0x7fb2b406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb2b406b860 0x7fb2b406dd20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009b40 tx=0x7fb2c4009190 comp rx=0 tx=0).stop 2024-03-23T15:57:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 msgr2=0x7fb2d0153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 0x7fb2d0153620 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c6b0 tx=0x7fb2c800cb70 comp rx=0 tx=0).stop 2024-03-23T15:57:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 shutdown_connections 2024-03-23T15:57:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb2b406b860 0x7fb2b406dd20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2d00cc080 0x7fb2d0153620 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2d00cbc60 0x7fb2d01f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 --2- 172.21.15.129:0/326359950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d006c030 0x7fb2d014db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.539+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 >> 172.21.15.129:0/326359950 conn(0x7fb2d00cfb20 msgr2=0x7fb2d01590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.540+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 shutdown_connections 2024-03-23T15:57:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:31.540+0000 7fb2d6cc3640 1 -- 172.21.15.129:0/326359950 wait complete. 2024-03-23T15:57:32.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:32.157 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:33.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 Processor -- start 2024-03-23T15:57:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 -- start start 2024-03-23T15:57:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8cf406c030 0x7f8cf406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 0x7f8cf415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cf406cae0 con 0x7f8cf40cbc60 2024-03-23T15:57:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8cf40cabc0 con 0x7f8cf406c030 2024-03-23T15:57:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.258+0000 7f8cfabf4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8cf40caea0 con 0x7f8cf40cc080 2024-03-23T15:57:33.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.259+0000 7f8cf8990640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8cf406c030 0x7f8cf406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:33.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.259+0000 7f8cf9191640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 0x7f8cf415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:33.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.260+0000 7f8cf9191640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 0x7f8cf415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44848/0 (socket says 172.21.15.129:44848) 2024-03-23T15:57:33.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.260+0000 7f8cf9191640 1 -- 172.21.15.129:0/340257009 learned_addr learned my addr 172.21.15.129:0/340257009 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.262+0000 7f8cf9191640 1 -- 172.21.15.129:0/340257009 >> v2:172.21.15.129:3300/0 conn(0x7f8cf406c030 msgr2=0x7f8cf406c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.262+0000 7f8cf9191640 1 --2- 172.21.15.129:0/340257009 >> v2:172.21.15.129:3300/0 conn(0x7f8cf406c030 0x7f8cf406c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.262+0000 7f8cf9191640 1 -- 172.21.15.129:0/340257009 >> v1:172.21.15.129:6789/0 conn(0x7f8cf40cbc60 legacy=0x7f8cf415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.262+0000 7f8cf9191640 1 -- 172.21.15.129:0/340257009 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf40cb180 con 0x7f8cf40cc080 2024-03-23T15:57:33.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.264+0000 7f8cf9191640 1 --2- 172.21.15.129:0/340257009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 0x7f8cf415e010 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009e50 tx=0x7f8cec02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4200f97c93f41dca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.267+0000 7f8cf37fe640 1 -- 172.21.15.129:0/340257009 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec02ca70 con 0x7f8cf40cc080 2024-03-23T15:57:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cf37fe640 1 -- 172.21.15.129:0/340257009 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cec02cbd0 con 0x7f8cf40cc080 2024-03-23T15:57:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cf37fe640 1 -- 172.21.15.129:0/340257009 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0343f0 con 0x7f8cf40cc080 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 -- 172.21.15.129:0/340257009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 msgr2=0x7f8cf415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/340257009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 0x7f8cf415e010 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009e50 tx=0x7f8cec02bed0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 -- 172.21.15.129:0/340257009 shutdown_connections 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/340257009 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cc080 0x7f8cf415e010 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/340257009 >> v2:172.21.15.129:3300/0 conn(0x7f8cf406c030 0x7f8cf406c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 -- 172.21.15.129:0/340257009 >> 172.21.15.129:0/340257009 conn(0x7f8cf40cfb20 msgr2=0x7f8cf40cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 -- 172.21.15.129:0/340257009 shutdown_connections 2024-03-23T15:57:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.269+0000 7f8cfabf4640 1 -- 172.21.15.129:0/340257009 wait complete. 2024-03-23T15:57:33.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 Processor -- start 2024-03-23T15:57:33.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 -- start start 2024-03-23T15:57:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cf406c030 0x7f8cf415cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cbc60 0x7f8cf41e7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 0x7f8cf415df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf40c5560 con 0x7f8cf40cc080 2024-03-23T15:57:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf415dab0 con 0x7f8cf406c030 2024-03-23T15:57:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.271+0000 7f8cfabf4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8cf41f0650 con 0x7f8cf40cbc60 2024-03-23T15:57:33.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.272+0000 7f8cf3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cbc60 0x7f8cf41e7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:33.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.272+0000 7f8cf9191640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 0x7f8cf415df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:33.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.272+0000 7f8cf8990640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cf406c030 0x7f8cf415cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:33.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.273+0000 7f8cf9191640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 0x7f8cf415df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52394/0 (socket says 172.21.15.129:52394) 2024-03-23T15:57:33.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.273+0000 7f8cf9191640 1 -- 172.21.15.129:0/2344395310 learned_addr learned my addr 172.21.15.129:0/2344395310 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:33.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.276+0000 7f8cf9191640 1 -- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cbc60 msgr2=0x7f8cf41e7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:33.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.276+0000 7f8cf9191640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cbc60 0x7f8cf41e7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.276+0000 7f8cf9191640 1 -- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cf406c030 msgr2=0x7f8cf415cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:33.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.276+0000 7f8cf9191640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cf406c030 0x7f8cf415cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.276+0000 7f8cf9191640 1 -- 172.21.15.129:0/2344395310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cec009b30 con 0x7f8cf40cc080 2024-03-23T15:57:33.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.279+0000 7f8cf9191640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 0x7f8cf415df40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009e50 tx=0x7f8cec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:33.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.284+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec009510 con 0x7f8cf40cc080 2024-03-23T15:57:33.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.284+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cec03e450 con 0x7f8cf40cc080 2024-03-23T15:57:33.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.284+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec03e820 con 0x7f8cf40cc080 2024-03-23T15:57:33.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.284+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf40c49e0 con 0x7f8cf40cc080 2024-03-23T15:57:33.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.284+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf40c4fa0 con 0x7f8cf40cc080 2024-03-23T15:57:33.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.285+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf415d880 con 0x7f8cf40cc080 2024-03-23T15:57:33.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.538+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8cec03e980 con 0x7f8cf40cc080 2024-03-23T15:57:33.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.538+0000 7f8cf17fa640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8cd806b810 0x7f8cd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:33.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.538+0000 7f8cf8990640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8cd806b810 0x7f8cd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:33.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.539+0000 7f8cf8990640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8cd806b810 0x7f8cd806dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009b40 tx=0x7f8ce8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:33.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.552+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8cec07b870 con 0x7f8cf40cc080 2024-03-23T15:57:33.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.552+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8cec077b10 con 0x7f8cf40cc080 2024-03-23T15:57:33.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.771+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cf415dc60 con 0x7f8cf40cc080 2024-03-23T15:57:33.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.792+0000 7f8cf17fa640 1 -- 172.21.15.129:0/2344395310 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8cf415dc60 con 0x7f8cf40cc080 2024-03-23T15:57:33.795 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:33.795 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:33.795 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8cd806b810 msgr2=0x7f8cd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8cd806b810 0x7f8cd806dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009b40 tx=0x7f8ce8009190 comp rx=0 tx=0).stop 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 msgr2=0x7f8cf415df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 0x7f8cf415df40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009e50 tx=0x7f8cec004a40 comp rx=0 tx=0).stop 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 shutdown_connections 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cf40cc080 0x7f8cf415df40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8cf40cbc60 0x7f8cf41e7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8cd806b810 0x7f8cd806dcd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 --2- 172.21.15.129:0/2344395310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8cf406c030 0x7f8cf415cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 >> 172.21.15.129:0/2344395310 conn(0x7f8cf40cfb20 msgr2=0x7f8cf40cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 shutdown_connections 2024-03-23T15:57:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:33.794+0000 7f8cfabf4640 1 -- 172.21.15.129:0/2344395310 wait complete. 2024-03-23T15:57:34.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:34.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:35.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:35.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.513+0000 7f42f84ed640 1 Processor -- start 2024-03-23T15:57:35.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.513+0000 7f42f84ed640 1 -- start start 2024-03-23T15:57:35.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.514+0000 7f42f84ed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 0x7f42f01568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.514+0000 7f42f84ed640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f42f0156df0 0x7f42f006d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.514+0000 7f42f84ed640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f0073560 con 0x7f42f0072ad0 2024-03-23T15:57:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.514+0000 7f42f84ed640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f42f01510e0 con 0x7f42f0156df0 2024-03-23T15:57:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.514+0000 7f42f84ed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f42f0072fc0 con 0x7f42f0150d00 2024-03-23T15:57:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.515+0000 7f42f6289640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 0x7f42f01568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:35.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.515+0000 7f42f6289640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 0x7f42f01568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:44880/0 (socket says 172.21.15.129:44880) 2024-03-23T15:57:35.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.515+0000 7f42f6289640 1 -- 172.21.15.129:0/204191332 learned_addr learned my addr 172.21.15.129:0/204191332 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:35.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.516+0000 7f42f5a88640 1 --2- 172.21.15.129:0/204191332 >> v2:172.21.15.157:3300/0 conn(0x7f42f0156df0 0x7f42f006d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:35.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.517+0000 7f42f6289640 1 -- 172.21.15.129:0/204191332 >> v2:172.21.15.157:3300/0 conn(0x7f42f0156df0 msgr2=0x7f42f006d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:35.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.517+0000 7f42f6289640 1 --2- 172.21.15.129:0/204191332 >> v2:172.21.15.157:3300/0 conn(0x7f42f0156df0 0x7f42f006d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:35.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.517+0000 7f42f6289640 1 -- 172.21.15.129:0/204191332 >> v1:172.21.15.157:6789/0 conn(0x7f42f0072ad0 legacy=0x7f42f0159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:35.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.517+0000 7f42f6289640 1 -- 172.21.15.129:0/204191332 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f0073280 con 0x7f42f0150d00 2024-03-23T15:57:35.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.520+0000 7f42f6289640 1 --2- 172.21.15.129:0/204191332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 0x7f42f01568b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00bb10 tx=0x7f42ec02e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282487f14c204b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:35.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.524+0000 7f42f5287640 1 -- 172.21.15.129:0/204191332 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec036440 con 0x7f42f0150d00 2024-03-23T15:57:35.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.524+0000 7f42f5287640 1 -- 172.21.15.129:0/204191332 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ec0365a0 con 0x7f42f0150d00 2024-03-23T15:57:35.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.524+0000 7f42f5287640 1 -- 172.21.15.129:0/204191332 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec036930 con 0x7f42f0150d00 2024-03-23T15:57:35.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 -- 172.21.15.129:0/204191332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 msgr2=0x7f42f01568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:35.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 --2- 172.21.15.129:0/204191332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 0x7f42f01568b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00bb10 tx=0x7f42ec02e150 comp rx=0 tx=0).stop 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 -- 172.21.15.129:0/204191332 shutdown_connections 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 --2- 172.21.15.129:0/204191332 >> v2:172.21.15.157:3300/0 conn(0x7f42f0156df0 0x7f42f006d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 --2- 172.21.15.129:0/204191332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0150d00 0x7f42f01568b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 -- 172.21.15.129:0/204191332 >> 172.21.15.129:0/204191332 conn(0x7f42f00c8370 msgr2=0x7f42f00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 -- 172.21.15.129:0/204191332 shutdown_connections 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.525+0000 7f42f84ed640 1 -- 172.21.15.129:0/204191332 wait complete. 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.526+0000 7f42f84ed640 1 Processor -- start 2024-03-23T15:57:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.526+0000 7f42f84ed640 1 -- start start 2024-03-23T15:57:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f84ed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42f0072ad0 0x7f42f0146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f84ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42f0150d00 0x7f42f01eb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f84ed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0156df0 0x7f42f0071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f84ed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01ef830 con 0x7f42f0150d00 2024-03-23T15:57:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f84ed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f42f01473c0 con 0x7f42f0072ad0 2024-03-23T15:57:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f84ed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f42f014eb10 con 0x7f42f0156df0 2024-03-23T15:57:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.527+0000 7f42f5a88640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42f0150d00 0x7f42f01eb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.528+0000 7f42f6289640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42f0072ad0 0x7f42f0146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.528+0000 7f42f6a8a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0156df0 0x7f42f0071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.528+0000 7f42f6289640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42f0072ad0 0x7f42f0146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38548/0 (socket says 172.21.15.129:38548) 2024-03-23T15:57:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.528+0000 7f42f6289640 1 -- 172.21.15.129:0/3065031238 learned_addr learned my addr 172.21.15.129:0/3065031238 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:35.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.531+0000 7f42f5a88640 1 -- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0156df0 msgr2=0x7f42f0071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:35.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.531+0000 7f42f5a88640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0156df0 0x7f42f0071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:35.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.531+0000 7f42f5a88640 1 -- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42f0072ad0 msgr2=0x7f42f0146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:35.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.531+0000 7f42f5a88640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42f0072ad0 0x7f42f0146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:35.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.531+0000 7f42f5a88640 1 -- 172.21.15.129:0/3065031238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec00b770 con 0x7f42f0150d00 2024-03-23T15:57:35.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.533+0000 7f42f5a88640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42f0150d00 0x7f42f01eb920 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f42e000e870 tx=0x7f42e000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.538+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e000ee60 con 0x7f42f0150d00 2024-03-23T15:57:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.538+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e000c940 con 0x7f42f0150d00 2024-03-23T15:57:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.538+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f01eee90 con 0x7f42f0150d00 2024-03-23T15:57:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.538+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e000ccd0 con 0x7f42f0150d00 2024-03-23T15:57:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.538+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f01ef450 con 0x7f42f0150d00 2024-03-23T15:57:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.539+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f01efaa0 con 0x7f42f0150d00 2024-03-23T15:57:35.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.561+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42e0013470 con 0x7f42f0150d00 2024-03-23T15:57:35.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.561+0000 7f42e6ffd640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42d406b810 0x7f42d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:35.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.561+0000 7f42f6289640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42d406b810 0x7f42d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:35.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.562+0000 7f42f6289640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42d406b810 0x7f42d406dcd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f42ec039040 tx=0x7f42ec032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.564+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f42e0014e40 con 0x7f42f0150d00 2024-03-23T15:57:35.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:35.837+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42e0055410 con 0x7f42f0150d00 2024-03-23T15:57:36.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.059+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42f0075740 con 0x7f42f0150d00 2024-03-23T15:57:36.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.080+0000 7f42e6ffd640 1 -- 172.21.15.129:0/3065031238 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f42e0021090 con 0x7f42f0150d00 2024-03-23T15:57:36.082 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:36.082 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:36.083 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42d406b810 msgr2=0x7f42d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42d406b810 0x7f42d406dcd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f42ec039040 tx=0x7f42ec032000 comp rx=0 tx=0).stop 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42f0150d00 msgr2=0x7f42f01eb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42f0150d00 0x7f42f01eb920 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f42e000e870 tx=0x7f42e000ed30 comp rx=0 tx=0).stop 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 shutdown_connections 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42f0156df0 0x7f42f0071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42f0150d00 0x7f42f01eb920 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f42d406b810 0x7f42d406dcd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 --2- 172.21.15.129:0/3065031238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42f0072ad0 0x7f42f0146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 >> 172.21.15.129:0/3065031238 conn(0x7f42f00c8370 msgr2=0x7f42f0149e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 shutdown_connections 2024-03-23T15:57:36.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:36.082+0000 7f42f84ed640 1 -- 172.21.15.129:0/3065031238 wait complete. 2024-03-23T15:57:36.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:36.699 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:37.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:37.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 Processor -- start 2024-03-23T15:57:37.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 -- start start 2024-03-23T15:57:37.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f198c06c030 0x7f198c06c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c0cbc60 0x7f198c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f198c0cc080 0x7f198c15de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f198c06cae0 con 0x7f198c0cc080 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f198c0cabc0 con 0x7f198c06c030 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.801+0000 7f1991c07640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f198c0caea0 con 0x7f198c0cbc60 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.802+0000 7f198b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f198c06c030 0x7f198c06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.802+0000 7f198affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c0cbc60 0x7f198c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.802+0000 7f198bfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f198c0cc080 0x7f198c15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.802+0000 7f198b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f198c06c030 0x7f198c06c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38552/0 (socket says 172.21.15.129:38552) 2024-03-23T15:57:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.802+0000 7f198b7fe640 1 -- 172.21.15.129:0/2924386915 learned_addr learned my addr 172.21.15.129:0/2924386915 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.805+0000 7f198affd640 1 -- 172.21.15.129:0/2924386915 >> v2:172.21.15.157:3300/0 conn(0x7f198c06c030 msgr2=0x7f198c06c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.805+0000 7f198affd640 1 --2- 172.21.15.129:0/2924386915 >> v2:172.21.15.157:3300/0 conn(0x7f198c06c030 0x7f198c06c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.805+0000 7f198affd640 1 -- 172.21.15.129:0/2924386915 >> v2:172.21.15.129:3300/0 conn(0x7f198c0cc080 msgr2=0x7f198c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.805+0000 7f198affd640 1 --2- 172.21.15.129:0/2924386915 >> v2:172.21.15.129:3300/0 conn(0x7f198c0cc080 0x7f198c15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.805+0000 7f198affd640 1 -- 172.21.15.129:0/2924386915 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c0cb180 con 0x7f198c0cbc60 2024-03-23T15:57:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.805+0000 7f198bfff640 1 --2- 172.21.15.129:0/2924386915 >> v2:172.21.15.129:3300/0 conn(0x7f198c0cc080 0x7f198c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:57:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.807+0000 7f198affd640 1 --2- 172.21.15.129:0/2924386915 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c0cbc60 0x7f198c15d900 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f198c0cf770 tx=0x7f198002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8673945282516cce server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:37.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.811+0000 7f198a7fc640 1 -- 172.21.15.129:0/2924386915 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198002f050 con 0x7f198c0cbc60 2024-03-23T15:57:37.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f198a7fc640 1 -- 172.21.15.129:0/2924386915 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1980031700 con 0x7f198c0cbc60 2024-03-23T15:57:37.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f198a7fc640 1 -- 172.21.15.129:0/2924386915 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980031a90 con 0x7f198c0cbc60 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 -- 172.21.15.129:0/2924386915 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c0cbc60 msgr2=0x7f198c15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 --2- 172.21.15.129:0/2924386915 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c0cbc60 0x7f198c15d900 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f198c0cf770 tx=0x7f198002b7a0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 -- 172.21.15.129:0/2924386915 shutdown_connections 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 --2- 172.21.15.129:0/2924386915 >> v2:172.21.15.129:3300/0 conn(0x7f198c0cc080 0x7f198c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 --2- 172.21.15.129:0/2924386915 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c0cbc60 0x7f198c15d900 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 --2- 172.21.15.129:0/2924386915 >> v2:172.21.15.157:3300/0 conn(0x7f198c06c030 0x7f198c06c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 -- 172.21.15.129:0/2924386915 >> 172.21.15.129:0/2924386915 conn(0x7f198c0cfb20 msgr2=0x7f198c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:37.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 -- 172.21.15.129:0/2924386915 shutdown_connections 2024-03-23T15:57:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.812+0000 7f1991c07640 1 -- 172.21.15.129:0/2924386915 wait complete. 2024-03-23T15:57:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 Processor -- start 2024-03-23T15:57:37.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 -- start start 2024-03-23T15:57:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c06c030 0x7f198c1ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 0x7f198c150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f198c0cc080 0x7f198c156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f198c1f36e0 con 0x7f198c0cbc60 2024-03-23T15:57:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f198c1e4120 con 0x7f198c0cc080 2024-03-23T15:57:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.814+0000 7f1991c07640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f198c0c3250 con 0x7f198c06c030 2024-03-23T15:57:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.815+0000 7f198affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 0x7f198c150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.815+0000 7f198b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c06c030 0x7f198c1ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.815+0000 7f198bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f198c0cc080 0x7f198c156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.815+0000 7f198affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 0x7f198c150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52420/0 (socket says 172.21.15.129:52420) 2024-03-23T15:57:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.815+0000 7f198affd640 1 -- 172.21.15.129:0/3335127444 learned_addr learned my addr 172.21.15.129:0/3335127444 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.819+0000 7f198affd640 1 -- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c06c030 msgr2=0x7f198c1ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.819+0000 7f198affd640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c06c030 0x7f198c1ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.819+0000 7f198affd640 1 -- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f198c0cc080 msgr2=0x7f198c156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.819+0000 7f198affd640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f198c0cc080 0x7f198c156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.819+0000 7f198affd640 1 -- 172.21.15.129:0/3335127444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1984009a70 con 0x7f198c0cbc60 2024-03-23T15:57:37.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.821+0000 7f198affd640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 0x7f198c150150 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1980000c00 tx=0x7f19800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.823+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980006e10 con 0x7f198c0cbc60 2024-03-23T15:57:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.823+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1980009400 con 0x7f198c0cbc60 2024-03-23T15:57:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.824+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c1e3e40 con 0x7f198c0cbc60 2024-03-23T15:57:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.824+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198c075740 con 0x7f198c0cbc60 2024-03-23T15:57:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.826+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1980033a30 con 0x7f198c0cbc60 2024-03-23T15:57:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.826+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198002f020 con 0x7f198c0cbc60 2024-03-23T15:57:37.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.849+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f198003a040 con 0x7f198c0cbc60 2024-03-23T15:57:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.849+0000 7f1973fff640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f197806b810 0x7f197806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.849+0000 7f198b7fe640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f197806b810 0x7f197806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:37.850+0000 7f198b7fe640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f197806b810 0x7f197806dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f197c009b30 tx=0x7f197c009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:38.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.124+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f198007afe0 con 0x7f198c0cbc60 2024-03-23T15:57:38.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.124+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1980076b80 con 0x7f198c0cbc60 2024-03-23T15:57:38.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.348+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f198c0af230 con 0x7f198c0cbc60 2024-03-23T15:57:38.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.361+0000 7f1973fff640 1 -- 172.21.15.129:0/3335127444 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1980041090 con 0x7f198c0cbc60 2024-03-23T15:57:38.364 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:38.364 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:38.364 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.364+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f197806b810 msgr2=0x7f197806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.364+0000 7f1991c07640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f197806b810 0x7f197806dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f197c009b30 tx=0x7f197c009240 comp rx=0 tx=0).stop 2024-03-23T15:57:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.364+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 msgr2=0x7f198c150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.364+0000 7f1991c07640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 0x7f198c150150 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1980000c00 tx=0x7f19800046c0 comp rx=0 tx=0).stop 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 shutdown_connections 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f197806b810 0x7f197806dcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f198c0cc080 0x7f198c156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f198c0cbc60 0x7f198c150150 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 --2- 172.21.15.129:0/3335127444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f198c06c030 0x7f198c1ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 >> 172.21.15.129:0/3335127444 conn(0x7f198c0cfb20 msgr2=0x7f198c0ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 shutdown_connections 2024-03-23T15:57:38.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:38.365+0000 7f1991c07640 1 -- 172.21.15.129:0/3335127444 wait complete. 2024-03-23T15:57:38.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:57:38.640 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:57:38.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:38.987 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:39.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:40.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.089+0000 7f6a230df640 1 Processor -- start 2024-03-23T15:57:40.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.089+0000 7f6a230df640 1 -- start start 2024-03-23T15:57:40.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.090+0000 7f6a230df640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6a1c06c030 0x7f6a1c06c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.090+0000 7f6a230df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a1c0cbc60 0x7f6a1c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.090+0000 7f6a230df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.090+0000 7f6a230df640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6a1c06cae0 con 0x7f6a1c06c030 2024-03-23T15:57:40.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.090+0000 7f6a230df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6a1c0cabc0 con 0x7f6a1c0cbc60 2024-03-23T15:57:40.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.090+0000 7f6a230df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a1c0caea0 con 0x7f6a1c0cc080 2024-03-23T15:57:40.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.091+0000 7f6a1bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a1c0cbc60 0x7f6a1c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.091+0000 7f6a20e7b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6a1c06c030 0x7f6a1c06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.091+0000 7f6a2167c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.091+0000 7f6a1bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a1c0cbc60 0x7f6a1c15d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38570/0 (socket says 172.21.15.129:38570) 2024-03-23T15:57:40.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.091+0000 7f6a1bfff640 1 -- 172.21.15.129:0/497426685 learned_addr learned my addr 172.21.15.129:0/497426685 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.094+0000 7f6a20e7b640 1 -- 172.21.15.129:0/497426685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 msgr2=0x7f6a1c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.094+0000 7f6a20e7b640 1 --2- 172.21.15.129:0/497426685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.094+0000 7f6a20e7b640 1 -- 172.21.15.129:0/497426685 >> v2:172.21.15.157:3300/0 conn(0x7f6a1c0cbc60 msgr2=0x7f6a1c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.094+0000 7f6a20e7b640 1 --2- 172.21.15.129:0/497426685 >> v2:172.21.15.157:3300/0 conn(0x7f6a1c0cbc60 0x7f6a1c15d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.094+0000 7f6a20e7b640 1 -- 172.21.15.129:0/497426685 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1c0cb180 con 0x7f6a1c06c030 2024-03-23T15:57:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.096+0000 7f6a20e7b640 1 --2- 172.21.15.129:0/497426685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c06c030 0x7f6a1c06c410 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0cd660 tx=0x7f6a1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a54238feab14f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.099+0000 7f6a1b7fe640 1 -- 172.21.15.129:0/497426685 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1002e070 con 0x7f6a1c06c030 2024-03-23T15:57:40.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.099+0000 7f6a1b7fe640 1 -- 172.21.15.129:0/497426685 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a100315a0 con 0x7f6a1c06c030 2024-03-23T15:57:40.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.099+0000 7f6a1b7fe640 1 -- 172.21.15.129:0/497426685 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a10031930 con 0x7f6a1c06c030 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 -- 172.21.15.129:0/497426685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c06c030 msgr2=0x7f6a1c06c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 --2- 172.21.15.129:0/497426685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c06c030 0x7f6a1c06c410 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0cd660 tx=0x7f6a1002b840 comp rx=0 tx=0).stop 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 -- 172.21.15.129:0/497426685 shutdown_connections 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 --2- 172.21.15.129:0/497426685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 --2- 172.21.15.129:0/497426685 >> v2:172.21.15.157:3300/0 conn(0x7f6a1c0cbc60 0x7f6a1c15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 --2- 172.21.15.129:0/497426685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c06c030 0x7f6a1c06c410 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 -- 172.21.15.129:0/497426685 >> 172.21.15.129:0/497426685 conn(0x7f6a1c0cfb20 msgr2=0x7f6a1c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:40.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 -- 172.21.15.129:0/497426685 shutdown_connections 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.100+0000 7f6a230df640 1 -- 172.21.15.129:0/497426685 wait complete. 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.101+0000 7f6a230df640 1 Processor -- start 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.101+0000 7f6a230df640 1 -- start start 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.102+0000 7f6a230df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a1c06c030 0x7f6a1c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.102+0000 7f6a230df640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c0cbc60 0x7f6a1c15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.102+0000 7f6a230df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.102+0000 7f6a230df640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c157730 con 0x7f6a1c0cbc60 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.102+0000 7f6a230df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c1f3c60 con 0x7f6a1c06c030 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.102+0000 7f6a230df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a1c1582a0 con 0x7f6a1c0cc080 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.103+0000 7f6a20e7b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a1c06c030 0x7f6a1c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.103+0000 7f6a2167c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.103+0000 7f6a1bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c0cbc60 0x7f6a1c15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.103+0000 7f6a20e7b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a1c06c030 0x7f6a1c159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38582/0 (socket says 172.21.15.129:38582) 2024-03-23T15:57:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.103+0000 7f6a20e7b640 1 -- 172.21.15.129:0/2167579361 learned_addr learned my addr 172.21.15.129:0/2167579361 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:40.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.107+0000 7f6a2167c640 1 -- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a1c06c030 msgr2=0x7f6a1c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:40.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.107+0000 7f6a2167c640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a1c06c030 0x7f6a1c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.107+0000 7f6a2167c640 1 -- 172.21.15.129:0/2167579361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c0cbc60 msgr2=0x7f6a1c15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:40.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.107+0000 7f6a2167c640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c0cbc60 0x7f6a1c15d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.107+0000 7f6a2167c640 1 -- 172.21.15.129:0/2167579361 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a10009400 con 0x7f6a1c0cc080 2024-03-23T15:57:40.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.109+0000 7f6a2167c640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15dd30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400eba0 tx=0x7f6a1400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.113+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1400cd20 con 0x7f6a1c0cc080 2024-03-23T15:57:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.114+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1400f040 con 0x7f6a1c0cc080 2024-03-23T15:57:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.114+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1c158580 con 0x7f6a1c0cc080 2024-03-23T15:57:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.114+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a140146c0 con 0x7f6a1c0cc080 2024-03-23T15:57:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.114+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a1c158b40 con 0x7f6a1c0cc080 2024-03-23T15:57:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.115+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a1c15d510 con 0x7f6a1c0cc080 2024-03-23T15:57:40.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.358+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6a14014820 con 0x7f6a1c0cc080 2024-03-23T15:57:40.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.358+0000 7f6a197fa640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a0006b890 0x7f6a0006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:40.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.359+0000 7f6a20e7b640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a0006b890 0x7f6a0006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:40.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.359+0000 7f6a20e7b640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a0006b890 0x7f6a0006dd50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6a1002b880 tx=0x7f6a1003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:40.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.391+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6a14059960 con 0x7f6a1c0cc080 2024-03-23T15:57:40.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.409+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a14055c00 con 0x7f6a1c0cc080 2024-03-23T15:57:40.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.632+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a1c15d510 con 0x7f6a1c0cc080 2024-03-23T15:57:40.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.789+0000 7f6a197fa640 1 -- 172.21.15.129:0/2167579361 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6a1c15d510 con 0x7f6a1c0cc080 2024-03-23T15:57:40.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:40.792 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:40.792 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:40.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a0006b890 msgr2=0x7f6a0006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:40.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a0006b890 0x7f6a0006dd50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6a1002b880 tx=0x7f6a1003d040 comp rx=0 tx=0).stop 2024-03-23T15:57:40.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 msgr2=0x7f6a1c15dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:40.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15dd30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400eba0 tx=0x7f6a1400c490 comp rx=0 tx=0).stop 2024-03-23T15:57:40.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 shutdown_connections 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a1c0cc080 0x7f6a1c15dd30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a1c0cbc60 0x7f6a1c15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6a0006b890 0x7f6a0006dd50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 --2- 172.21.15.129:0/2167579361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a1c06c030 0x7f6a1c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 >> 172.21.15.129:0/2167579361 conn(0x7f6a1c0cfb20 msgr2=0x7f6a1c154d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 shutdown_connections 2024-03-23T15:57:40.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:40.792+0000 7f6a230df640 1 -- 172.21.15.129:0/2167579361 wait complete. 2024-03-23T15:57:41.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:41.410 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:42.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:42.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.511+0000 7fdb18325640 1 Processor -- start 2024-03-23T15:57:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.511+0000 7fdb18325640 1 -- start start 2024-03-23T15:57:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.512+0000 7fdb18325640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdb101544b0 0x7fdb1015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.512+0000 7fdb18325640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdb100caa00 0x7fdb1006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.512+0000 7fdb18325640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb100cbcb0 con 0x7fdb1006c0b0 2024-03-23T15:57:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.512+0000 7fdb18325640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fdb100c5be0 con 0x7fdb101544b0 2024-03-23T15:57:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.512+0000 7fdb18325640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdb1015b280 con 0x7fdb100caa00 2024-03-23T15:57:42.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.513+0000 7fdb160c1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdb101544b0 0x7fdb1015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:42.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.513+0000 7fdb168c2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fdb1006c0b0 0x7fdb100c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:40892/0 (socket says 172.21.15.129:40892) 2024-03-23T15:57:42.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.513+0000 7fdb168c2640 1 -- 172.21.15.129:0/3178856681 learned_addr learned my addr 172.21.15.129:0/3178856681 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:42.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.513+0000 7fdb158c0640 1 --2- 172.21.15.129:0/3178856681 >> v2:172.21.15.157:3300/0 conn(0x7fdb100caa00 0x7fdb1006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:42.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.515+0000 7fdb160c1640 1 -- 172.21.15.129:0/3178856681 >> v2:172.21.15.157:3300/0 conn(0x7fdb100caa00 msgr2=0x7fdb1006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:42.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.515+0000 7fdb160c1640 1 --2- 172.21.15.129:0/3178856681 >> v2:172.21.15.157:3300/0 conn(0x7fdb100caa00 0x7fdb1006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:42.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.515+0000 7fdb160c1640 1 -- 172.21.15.129:0/3178856681 >> v1:172.21.15.157:6789/0 conn(0x7fdb1006c0b0 legacy=0x7fdb100c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:42.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.516+0000 7fdb160c1640 1 -- 172.21.15.129:0/3178856681 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1015a9e0 con 0x7fdb101544b0 2024-03-23T15:57:42.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.518+0000 7fdb160c1640 1 --2- 172.21.15.129:0/3178856681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1015a2e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdb1006ce60 tx=0x7fdb0c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467c84ef180608fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:42.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.521+0000 7fdb150bf640 1 -- 172.21.15.129:0/3178856681 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c030610 con 0x7fdb101544b0 2024-03-23T15:57:42.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.521+0000 7fdb150bf640 1 -- 172.21.15.129:0/3178856681 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0c030770 con 0x7fdb101544b0 2024-03-23T15:57:42.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.521+0000 7fdb150bf640 1 -- 172.21.15.129:0/3178856681 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c030b00 con 0x7fdb101544b0 2024-03-23T15:57:42.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 -- 172.21.15.129:0/3178856681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 msgr2=0x7fdb1015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:42.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 --2- 172.21.15.129:0/3178856681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1015a2e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdb1006ce60 tx=0x7fdb0c02b960 comp rx=0 tx=0).stop 2024-03-23T15:57:42.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 -- 172.21.15.129:0/3178856681 shutdown_connections 2024-03-23T15:57:42.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 --2- 172.21.15.129:0/3178856681 >> v2:172.21.15.157:3300/0 conn(0x7fdb100caa00 0x7fdb1006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:42.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 --2- 172.21.15.129:0/3178856681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1015a2e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:42.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 -- 172.21.15.129:0/3178856681 >> 172.21.15.129:0/3178856681 conn(0x7fdb10061b50 msgr2=0x7fdb10061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:42.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 -- 172.21.15.129:0/3178856681 shutdown_connections 2024-03-23T15:57:42.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.522+0000 7fdb18325640 1 -- 172.21.15.129:0/3178856681 wait complete. 2024-03-23T15:57:42.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.523+0000 7fdb18325640 1 Processor -- start 2024-03-23T15:57:42.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 -- start start 2024-03-23T15:57:42.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb1006c0b0 0x7fdb10155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:42.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100caa00 0x7fdb10156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:42.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:42.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb101510d0 con 0x7fdb101544b0 2024-03-23T15:57:42.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdb10154bd0 con 0x7fdb100caa00 2024-03-23T15:57:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.524+0000 7fdb18325640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb101f3380 con 0x7fdb1006c0b0 2024-03-23T15:57:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.525+0000 7fdb158c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100caa00 0x7fdb10156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.525+0000 7fdb160c1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb1006c0b0 0x7fdb10155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.525+0000 7fdb168c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.525+0000 7fdb158c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100caa00 0x7fdb10156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:39958/0 (socket says 172.21.15.129:39958) 2024-03-23T15:57:42.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.525+0000 7fdb158c0640 1 -- 172.21.15.129:0/3150971761 learned_addr learned my addr 172.21.15.129:0/3150971761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:42.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.529+0000 7fdb168c2640 1 -- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb1006c0b0 msgr2=0x7fdb10155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:42.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.529+0000 7fdb168c2640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb1006c0b0 0x7fdb10155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:42.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.529+0000 7fdb168c2640 1 -- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100caa00 msgr2=0x7fdb10156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:42.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.529+0000 7fdb168c2640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100caa00 0x7fdb10156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:42.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.529+0000 7fdb168c2640 1 -- 172.21.15.129:0/3150971761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c009400 con 0x7fdb101544b0 2024-03-23T15:57:42.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.532+0000 7fdb168c2640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1014a430 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800caa0 tx=0x7fdb0800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:42.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.536+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0800ac00 con 0x7fdb101544b0 2024-03-23T15:57:42.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.536+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0800ad60 con 0x7fdb101544b0 2024-03-23T15:57:42.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.536+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb10154e20 con 0x7fdb101544b0 2024-03-23T15:57:42.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.536+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08012600 con 0x7fdb101544b0 2024-03-23T15:57:42.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.536+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb101553e0 con 0x7fdb101544b0 2024-03-23T15:57:42.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.537+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb10075740 con 0x7fdb101544b0 2024-03-23T15:57:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.556+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb08007070 con 0x7fdb101544b0 2024-03-23T15:57:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.557+0000 7fdb06ffd640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf406b810 0x7fdaf406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.557+0000 7fdb160c1640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf406b810 0x7fdaf406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.558+0000 7fdb160c1640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf406b810 0x7fdaf406dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c036000 tx=0x7fdb0c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.559+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdb08058d70 con 0x7fdb101544b0 2024-03-23T15:57:42.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:42.803+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdb08055010 con 0x7fdb101544b0 2024-03-23T15:57:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.056+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb10157d00 con 0x7fdb101544b0 2024-03-23T15:57:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.107+0000 7fdb06ffd640 1 -- 172.21.15.129:0/3150971761 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fdb08017090 con 0x7fdb101544b0 2024-03-23T15:57:43.110 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:43.110 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf406b810 msgr2=0x7fdaf406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf406b810 0x7fdaf406dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c036000 tx=0x7fdb0c03b040 comp rx=0 tx=0).stop 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 msgr2=0x7fdb1014a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1014a430 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800caa0 tx=0x7fdb0800cf60 comp rx=0 tx=0).stop 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 shutdown_connections 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb101544b0 0x7fdb1014a430 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdaf406b810 0x7fdaf406dcd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb100caa00 0x7fdb10156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 --2- 172.21.15.129:0/3150971761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb1006c0b0 0x7fdb10155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 >> 172.21.15.129:0/3150971761 conn(0x7fdb10061b50 msgr2=0x7fdb100c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 shutdown_connections 2024-03-23T15:57:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:43.109+0000 7fdb18325640 1 -- 172.21.15.129:0/3150971761 wait complete. 2024-03-23T15:57:43.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:43.727 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:44.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:44.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.829+0000 7f3e771d7640 1 Processor -- start 2024-03-23T15:57:44.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.829+0000 7f3e771d7640 1 -- start start 2024-03-23T15:57:44.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e771d7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e700ceb00 0x7f3e7015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:44.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e771d7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3e700cef40 0x7f3e7015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:44.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e771d7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7015b290 con 0x7f3e700ccd60 2024-03-23T15:57:44.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e771d7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3e700ce800 con 0x7f3e700ceb00 2024-03-23T15:57:44.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e771d7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3e70154a70 con 0x7f3e700cef40 2024-03-23T15:57:44.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e6ffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3e700ceb00 0x7f3e7015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:44.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.830+0000 7f3e75774640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3e700cef40 0x7f3e7015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:44.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.831+0000 7f3e75774640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3e700cef40 0x7f3e7015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:39966/0 (socket says 172.21.15.129:39966) 2024-03-23T15:57:44.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.831+0000 7f3e75774640 1 -- 172.21.15.129:0/1344210316 learned_addr learned my addr 172.21.15.129:0/1344210316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.833+0000 7f3e75774640 1 -- 172.21.15.129:0/1344210316 >> v2:172.21.15.129:3300/0 conn(0x7f3e700ceb00 msgr2=0x7f3e7015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.833+0000 7f3e75774640 1 --2- 172.21.15.129:0/1344210316 >> v2:172.21.15.129:3300/0 conn(0x7f3e700ceb00 0x7f3e7015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.833+0000 7f3e75774640 1 -- 172.21.15.129:0/1344210316 >> v1:172.21.15.129:6789/0 conn(0x7f3e700ccd60 legacy=0x7f3e7006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.833+0000 7f3e75774640 1 -- 172.21.15.129:0/1344210316 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e700cd950 con 0x7f3e700cef40 2024-03-23T15:57:44.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.835+0000 7f3e75774640 1 --2- 172.21.15.129:0/1344210316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700cef40 0x7f3e7015b8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3e700cf660 tx=0x7f3e6802c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e205edde5e8bf61 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:44.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.838+0000 7f3e6f7fe640 1 -- 172.21.15.129:0/1344210316 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6802cbb0 con 0x7f3e700cef40 2024-03-23T15:57:44.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.840+0000 7f3e6f7fe640 1 -- 172.21.15.129:0/1344210316 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6802cd10 con 0x7f3e700cef40 2024-03-23T15:57:44.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.840+0000 7f3e6f7fe640 1 -- 172.21.15.129:0/1344210316 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68034610 con 0x7f3e700cef40 2024-03-23T15:57:44.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 -- 172.21.15.129:0/1344210316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700cef40 msgr2=0x7f3e7015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:44.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 --2- 172.21.15.129:0/1344210316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700cef40 0x7f3e7015b8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3e700cf660 tx=0x7f3e6802c010 comp rx=0 tx=0).stop 2024-03-23T15:57:44.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 -- 172.21.15.129:0/1344210316 shutdown_connections 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 --2- 172.21.15.129:0/1344210316 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700cef40 0x7f3e7015b8f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 --2- 172.21.15.129:0/1344210316 >> v2:172.21.15.129:3300/0 conn(0x7f3e700ceb00 0x7f3e7015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 -- 172.21.15.129:0/1344210316 >> 172.21.15.129:0/1344210316 conn(0x7f3e700b1840 msgr2=0x7f3e700b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 -- 172.21.15.129:0/1344210316 shutdown_connections 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.841+0000 7f3e771d7640 1 -- 172.21.15.129:0/1344210316 wait complete. 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.842+0000 7f3e771d7640 1 Processor -- start 2024-03-23T15:57:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.842+0000 7f3e771d7640 1 -- start start 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.843+0000 7f3e771d7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 0x7f3e701e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.843+0000 7f3e771d7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e700ceb00 0x7f3e700c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.843+0000 7f3e771d7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e700cef40 0x7f3e700c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.843+0000 7f3e771d7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700c7df0 con 0x7f3e700cef40 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.843+0000 7f3e771d7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700c7420 con 0x7f3e700ccd60 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.843+0000 7f3e771d7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e700c8720 con 0x7f3e700ceb00 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.844+0000 7f3e74f73640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 0x7f3e701e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.844+0000 7f3e75774640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e700cef40 0x7f3e700c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:44.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.844+0000 7f3e74f73640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 0x7f3e701e4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:39980/0 (socket says 172.21.15.129:39980) 2024-03-23T15:57:44.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.844+0000 7f3e74f73640 1 -- 172.21.15.129:0/731417729 learned_addr learned my addr 172.21.15.129:0/731417729 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:44.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.844+0000 7f3e6ffff640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e700ceb00 0x7f3e700c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.849+0000 7f3e74f73640 1 -- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e700ceb00 msgr2=0x7f3e700c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.849+0000 7f3e74f73640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e700ceb00 0x7f3e700c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.849+0000 7f3e74f73640 1 -- 172.21.15.129:0/731417729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e700cef40 msgr2=0x7f3e700c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.849+0000 7f3e74f73640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e700cef40 0x7f3e700c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.849+0000 7f3e74f73640 1 -- 172.21.15.129:0/731417729 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e68009b30 con 0x7f3e700ccd60 2024-03-23T15:57:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.849+0000 7f3e75774640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e700cef40 0x7f3e700c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:57:44.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.852+0000 7f3e74f73640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 0x7f3e701e4100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400c3f0 tx=0x7f3e6400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.856+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e640052d0 con 0x7f3e700ccd60 2024-03-23T15:57:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.856+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e64005450 con 0x7f3e700ccd60 2024-03-23T15:57:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.856+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e701f10a0 con 0x7f3e700ccd60 2024-03-23T15:57:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.856+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e64007260 con 0x7f3e700ccd60 2024-03-23T15:57:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.856+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7015d730 con 0x7f3e700ccd60 2024-03-23T15:57:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:44.857+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7004f530 con 0x7f3e700ccd60 2024-03-23T15:57:45.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.138+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e640073c0 con 0x7f3e700ccd60 2024-03-23T15:57:45.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.139+0000 7f3e6d7fa640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e5406b860 0x7f3e5406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:45.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.139+0000 7f3e6ffff640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e5406b860 0x7f3e5406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.139+0000 7f3e6ffff640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e5406b860 0x7f3e5406dd20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3e700b47c0 tx=0x7f3e5c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:45.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.144+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3e640583c0 con 0x7f3e700ccd60 2024-03-23T15:57:45.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.154+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e64054660 con 0x7f3e700ccd60 2024-03-23T15:57:45.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.380+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e7015b290 con 0x7f3e700ccd60 2024-03-23T15:57:45.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.461+0000 7f3e6d7fa640 1 -- 172.21.15.129:0/731417729 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3e7015b290 con 0x7f3e700ccd60 2024-03-23T15:57:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:45.462 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:45.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e5406b860 msgr2=0x7f3e5406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:45.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e5406b860 0x7f3e5406dd20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3e700b47c0 tx=0x7f3e5c009190 comp rx=0 tx=0).stop 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 msgr2=0x7f3e701e4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 0x7f3e701e4100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400c3f0 tx=0x7f3e6400c8b0 comp rx=0 tx=0).stop 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 shutdown_connections 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e700cef40 0x7f3e700c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e700ceb00 0x7f3e700c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e5406b860 0x7f3e5406dd20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 --2- 172.21.15.129:0/731417729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e700ccd60 0x7f3e701e4100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 >> 172.21.15.129:0/731417729 conn(0x7f3e700b1840 msgr2=0x7f3e700b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 shutdown_connections 2024-03-23T15:57:45.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:45.463+0000 7f3e771d7640 1 -- 172.21.15.129:0/731417729 wait complete. 2024-03-23T15:57:46.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:46.080 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:47.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:47.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.183+0000 7fc59b33e640 1 Processor -- start 2024-03-23T15:57:47.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.183+0000 7fc59b33e640 1 -- start start 2024-03-23T15:57:47.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.184+0000 7fc59b33e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc59406c030 0x7fc59406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:47.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.184+0000 7fc59b33e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59406cae0 con 0x7fc5940cbc60 2024-03-23T15:57:47.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.184+0000 7fc59b33e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5940cabc0 con 0x7fc5940cc080 2024-03-23T15:57:47.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.184+0000 7fc59b33e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc59415e720 con 0x7fc59406c030 2024-03-23T15:57:47.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.185+0000 7fc5990da640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc59406c030 0x7fc59406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:47.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.185+0000 7fc5998db640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc5940cc080 0x7fc59415e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:40898/0 (socket says 172.21.15.129:40898) 2024-03-23T15:57:47.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.185+0000 7fc5998db640 1 -- 172.21.15.129:0/3835055590 learned_addr learned my addr 172.21.15.129:0/3835055590 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.192+0000 7fc5990da640 1 -- 172.21.15.129:0/3835055590 >> v1:172.21.15.157:6789/0 conn(0x7fc5940cc080 legacy=0x7fc59415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.192+0000 7fc5990da640 1 -- 172.21.15.129:0/3835055590 >> v1:172.21.15.129:6789/0 conn(0x7fc5940cbc60 legacy=0x7fc59415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.192+0000 7fc5990da640 1 -- 172.21.15.129:0/3835055590 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5940caea0 con 0x7fc59406c030 2024-03-23T15:57:47.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.199+0000 7fc5990da640 1 --2- 172.21.15.129:0/3835055590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59406c030 0x7fc59406c410 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc5900097a0 tx=0x7fc59002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf34b2f23cd0a2eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:47.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.204+0000 7fc58bfff640 1 -- 172.21.15.129:0/3835055590 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc590031830 con 0x7fc59406c030 2024-03-23T15:57:47.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc58bfff640 1 -- 172.21.15.129:0/3835055590 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc590031990 con 0x7fc59406c030 2024-03-23T15:57:47.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc58bfff640 1 -- 172.21.15.129:0/3835055590 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc590031d20 con 0x7fc59406c030 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 -- 172.21.15.129:0/3835055590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59406c030 msgr2=0x7fc59406c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3835055590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59406c030 0x7fc59406c410 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc5900097a0 tx=0x7fc59002b8d0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 -- 172.21.15.129:0/3835055590 shutdown_connections 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3835055590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59406c030 0x7fc59406c410 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 -- 172.21.15.129:0/3835055590 >> 172.21.15.129:0/3835055590 conn(0x7fc5940cfb20 msgr2=0x7fc5940cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 -- 172.21.15.129:0/3835055590 shutdown_connections 2024-03-23T15:57:47.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.205+0000 7fc59b33e640 1 -- 172.21.15.129:0/3835055590 wait complete. 2024-03-23T15:57:47.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 Processor -- start 2024-03-23T15:57:47.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 -- start start 2024-03-23T15:57:47.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59406c030 0x7fc5941f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:47.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5940cbc60 0x7fc5941f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:47.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5940cc080 0x7fc59415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:47.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941553c0 con 0x7fc59406c030 2024-03-23T15:57:47.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc5941f1140 con 0x7fc5940cbc60 2024-03-23T15:57:47.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.207+0000 7fc59b33e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc5941594e0 con 0x7fc5940cc080 2024-03-23T15:57:47.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.208+0000 7fc5990da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59406c030 0x7fc5941f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:47.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.208+0000 7fc5998db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5940cc080 0x7fc59415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:47.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.208+0000 7fc5990da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59406c030 0x7fc5941f4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55030/0 (socket says 172.21.15.129:55030) 2024-03-23T15:57:47.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.208+0000 7fc5990da640 1 -- 172.21.15.129:0/3019694235 learned_addr learned my addr 172.21.15.129:0/3019694235 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:47.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.209+0000 7fc5988d9640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5940cbc60 0x7fc5941f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:47.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.212+0000 7fc5998db640 1 -- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5940cbc60 msgr2=0x7fc5941f15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:47.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.212+0000 7fc5998db640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5940cbc60 0x7fc5941f15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.212+0000 7fc5998db640 1 -- 172.21.15.129:0/3019694235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59406c030 msgr2=0x7fc5941f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:47.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.212+0000 7fc5998db640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59406c030 0x7fc5941f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.212+0000 7fc5998db640 1 -- 172.21.15.129:0/3019694235 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc590009400 con 0x7fc5940cc080 2024-03-23T15:57:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.215+0000 7fc5998db640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5940cc080 0x7fc59415a920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00caa0 tx=0x7fc58c00cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.219+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c00ab40 con 0x7fc5940cc080 2024-03-23T15:57:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.219+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58c00aca0 con 0x7fc5940cc080 2024-03-23T15:57:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.219+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5941f0e40 con 0x7fc5940cc080 2024-03-23T15:57:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.219+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c012600 con 0x7fc5940cc080 2024-03-23T15:57:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.219+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5941558b0 con 0x7fc5940cc080 2024-03-23T15:57:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.220+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5941f12f0 con 0x7fc5940cc080 2024-03-23T15:57:47.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.478+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc58c007070 con 0x7fc5940cc080 2024-03-23T15:57:47.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.478+0000 7fc589ffb640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc57806b890 0x7fc57806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:47.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.479+0000 7fc5990da640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc57806b890 0x7fc57806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:47.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.479+0000 7fc5990da640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc57806b890 0x7fc57806dd50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc59002b910 tx=0x7fc59003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:47.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.483+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc58c058b00 con 0x7fc5940cc080 2024-03-23T15:57:47.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.495+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc58c054da0 con 0x7fc5940cc080 2024-03-23T15:57:47.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.719+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5941f12f0 con 0x7fc5940cc080 2024-03-23T15:57:47.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.748+0000 7fc589ffb640 1 -- 172.21.15.129:0/3019694235 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc5941f12f0 con 0x7fc5940cc080 2024-03-23T15:57:47.751 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:47.751 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:47.752 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc57806b890 msgr2=0x7fc57806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc57806b890 0x7fc57806dd50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc59002b910 tx=0x7fc59003c040 comp rx=0 tx=0).stop 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5940cc080 msgr2=0x7fc59415a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5940cc080 0x7fc59415a920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00caa0 tx=0x7fc58c00cf60 comp rx=0 tx=0).stop 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 shutdown_connections 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5940cc080 0x7fc59415a920 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc57806b890 0x7fc57806dd50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5940cbc60 0x7fc5941f15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 --2- 172.21.15.129:0/3019694235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59406c030 0x7fc5941f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:47.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 >> 172.21.15.129:0/3019694235 conn(0x7fc5940cfb20 msgr2=0x7fc59415d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:47.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.751+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 shutdown_connections 2024-03-23T15:57:47.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:47.752+0000 7fc59b33e640 1 -- 172.21.15.129:0/3019694235 wait complete. 2024-03-23T15:57:48.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:48.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:49.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 Processor -- start 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 -- start start 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f60180ccd60 0x7f601815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180cef40 0x7f601815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601815b290 con 0x7f60180ceb00 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f60180ce800 con 0x7f60180ccd60 2024-03-23T15:57:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.472+0000 7f601e46f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6018154a70 con 0x7f60180cef40 2024-03-23T15:57:49.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.473+0000 7f6017fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f60180ccd60 0x7f601815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:49.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.473+0000 7f60177fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f60180ceb00 0x7f601806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:50010/0 (socket says 172.21.15.129:50010) 2024-03-23T15:57:49.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.473+0000 7f60177fe640 1 -- 172.21.15.129:0/1843443222 learned_addr learned my addr 172.21.15.129:0/1843443222 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:49.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.473+0000 7f601ca0c640 1 --2- 172.21.15.129:0/1843443222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180cef40 0x7f601815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:49.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.476+0000 7f601ca0c640 1 -- 172.21.15.129:0/1843443222 >> v2:172.21.15.157:3300/0 conn(0x7f60180ccd60 msgr2=0x7f601815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:49.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.476+0000 7f601ca0c640 1 --2- 172.21.15.129:0/1843443222 >> v2:172.21.15.157:3300/0 conn(0x7f60180ccd60 0x7f601815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:49.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.476+0000 7f601ca0c640 1 -- 172.21.15.129:0/1843443222 >> v1:172.21.15.129:6789/0 conn(0x7f60180ceb00 legacy=0x7f601806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:49.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.476+0000 7f601ca0c640 1 -- 172.21.15.129:0/1843443222 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60180cd950 con 0x7f60180cef40 2024-03-23T15:57:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.478+0000 7f601ca0c640 1 --2- 172.21.15.129:0/1843443222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180cef40 0x7f601815b8f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6010009d90 tx=0x7f601002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba97308d4da9248 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:49.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.482+0000 7f6016ffd640 1 -- 172.21.15.129:0/1843443222 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601002cce0 con 0x7f60180cef40 2024-03-23T15:57:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.485+0000 7f6016ffd640 1 -- 172.21.15.129:0/1843443222 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f601002ce40 con 0x7f60180cef40 2024-03-23T15:57:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.485+0000 7f6016ffd640 1 -- 172.21.15.129:0/1843443222 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010034660 con 0x7f60180cef40 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 -- 172.21.15.129:0/1843443222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180cef40 msgr2=0x7f601815b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 --2- 172.21.15.129:0/1843443222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180cef40 0x7f601815b8f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6010009d90 tx=0x7f601002bf30 comp rx=0 tx=0).stop 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 -- 172.21.15.129:0/1843443222 shutdown_connections 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 --2- 172.21.15.129:0/1843443222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180cef40 0x7f601815b8f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 --2- 172.21.15.129:0/1843443222 >> v2:172.21.15.157:3300/0 conn(0x7f60180ccd60 0x7f601815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 -- 172.21.15.129:0/1843443222 >> 172.21.15.129:0/1843443222 conn(0x7f60180b1840 msgr2=0x7f60180b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 -- 172.21.15.129:0/1843443222 shutdown_connections 2024-03-23T15:57:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.486+0000 7f601e46f640 1 -- 172.21.15.129:0/1843443222 wait complete. 2024-03-23T15:57:49.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.487+0000 7f601e46f640 1 Processor -- start 2024-03-23T15:57:49.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.487+0000 7f601e46f640 1 -- start start 2024-03-23T15:57:49.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601e46f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180ccd60 0x7f6018150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:49.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601e46f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 0x7f601815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601e46f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60180cef40 0x7f601815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601e46f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60180c93b0 con 0x7f60180ceb00 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601e46f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f60181f4300 con 0x7f60180cef40 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601e46f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f60181f3d60 con 0x7f60180ccd60 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f601ca0c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60180cef40 0x7f601815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.488+0000 7f60177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 0x7f601815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:49.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.489+0000 7f60177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 0x7f601815bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55042/0 (socket says 172.21.15.129:55042) 2024-03-23T15:57:49.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.489+0000 7f60177fe640 1 -- 172.21.15.129:0/2569141037 learned_addr learned my addr 172.21.15.129:0/2569141037 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:49.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.490+0000 7f6017fff640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180ccd60 0x7f6018150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:49.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.492+0000 7f60177fe640 1 -- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180ccd60 msgr2=0x7f6018150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:49.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.492+0000 7f60177fe640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180ccd60 0x7f6018150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:49.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.492+0000 7f60177fe640 1 -- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60180cef40 msgr2=0x7f601815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:49.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.492+0000 7f60177fe640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60180cef40 0x7f601815c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:49.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.492+0000 7f60177fe640 1 -- 172.21.15.129:0/2569141037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6010009a70 con 0x7f60180ceb00 2024-03-23T15:57:49.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.494+0000 7f60177fe640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 0x7f601815bcc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f600c00c400 tx=0x7f600c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:49.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.498+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c005ce0 con 0x7f60180ceb00 2024-03-23T15:57:49.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.498+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f600c005e40 con 0x7f60180ceb00 2024-03-23T15:57:49.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.498+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60180c6dd0 con 0x7f60180ceb00 2024-03-23T15:57:49.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.498+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c0072f0 con 0x7f60180ceb00 2024-03-23T15:57:49.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.498+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601815caf0 con 0x7f60180ceb00 2024-03-23T15:57:49.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.499+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601804f530 con 0x7f60180ceb00 2024-03-23T15:57:49.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.751+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f600c007510 con 0x7f60180ceb00 2024-03-23T15:57:49.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.752+0000 7f6014ff9640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f600406b810 0x7f600406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:49.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.752+0000 7f6017fff640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f600406b810 0x7f600406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:49.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.753+0000 7f6017fff640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f600406b810 0x7f600406dcd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6008009b40 tx=0x7f6008009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:49.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.765+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f600c058250 con 0x7f60180ceb00 2024-03-23T15:57:49.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:49.765+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f600c0544f0 con 0x7f60180ceb00 2024-03-23T15:57:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.015+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f601806db50 con 0x7f60180ceb00 2024-03-23T15:57:50.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.036+0000 7f6014ff9640 1 -- 172.21.15.129:0/2569141037 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f600c017030 con 0x7f60180ceb00 2024-03-23T15:57:50.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:50.038 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:50.039 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:50.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f600406b810 msgr2=0x7f600406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:50.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f600406b810 0x7f600406dcd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6008009b40 tx=0x7f6008009190 comp rx=0 tx=0).stop 2024-03-23T15:57:50.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 msgr2=0x7f601815bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:50.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 0x7f601815bcc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f600c00c400 tx=0x7f600c00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 shutdown_connections 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f600406b810 0x7f600406dcd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60180cef40 0x7f601815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60180ceb00 0x7f601815bcc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 --2- 172.21.15.129:0/2569141037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60180ccd60 0x7f6018150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 >> 172.21.15.129:0/2569141037 conn(0x7f60180b1840 msgr2=0x7f60180b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 shutdown_connections 2024-03-23T15:57:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:50.038+0000 7f601e46f640 1 -- 172.21.15.129:0/2569141037 wait complete. 2024-03-23T15:57:50.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:50.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:51.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:51.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.757+0000 7f450b15c640 1 Processor -- start 2024-03-23T15:57:51.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.757+0000 7f450b15c640 1 -- start start 2024-03-23T15:57:51.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.758+0000 7f450b15c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:51.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.758+0000 7f450b15c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4504070640 con 0x7f450415dba0 2024-03-23T15:57:51.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.758+0000 7f450b15c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45040c4c00 con 0x7f450406cf70 2024-03-23T15:57:51.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.758+0000 7f450b15c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f45040ccec0 con 0x7f45040cca30 2024-03-23T15:57:51.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.759+0000 7f4503fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f450406cf70 0x7f450415df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:52918/0 (socket says 172.21.15.129:52918) 2024-03-23T15:57:51.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.759+0000 7f4503fff640 1 -- 172.21.15.129:0/3558801670 learned_addr learned my addr 172.21.15.129:0/3558801670 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:51.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.759+0000 7f45096f9640 1 --2- 172.21.15.129:0/3558801670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:51.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.761+0000 7f45037fe640 1 -- 172.21.15.129:0/3558801670 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3209836569 0 0) 0x7f4504070640 con 0x7f450415dba0 2024-03-23T15:57:51.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.761+0000 7f45037fe640 1 -- 172.21.15.129:0/3558801670 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44f4003520 con 0x7f450415dba0 2024-03-23T15:57:51.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.762+0000 7f45096f9640 1 -- 172.21.15.129:0/3558801670 >> v1:172.21.15.157:6789/0 conn(0x7f450406cf70 legacy=0x7f450415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:51.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.762+0000 7f45096f9640 1 -- 172.21.15.129:0/3558801670 >> v1:172.21.15.129:6789/0 conn(0x7f450415dba0 legacy=0x7f450406ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:51.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.762+0000 7f45096f9640 1 -- 172.21.15.129:0/3558801670 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450406c240 con 0x7f45040cca30 2024-03-23T15:57:51.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.764+0000 7f45096f9640 1 --2- 172.21.15.129:0/3558801670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041540a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009df0 tx=0x7f44fc02bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acb61a019bf50748 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.767+0000 7f45037fe640 1 -- 172.21.15.129:0/3558801670 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc02cd40 con 0x7f45040cca30 2024-03-23T15:57:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.769+0000 7f45037fe640 1 -- 172.21.15.129:0/3558801670 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44fc034440 con 0x7f45040cca30 2024-03-23T15:57:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.769+0000 7f45037fe640 1 -- 172.21.15.129:0/3558801670 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc02cea0 con 0x7f45040cca30 2024-03-23T15:57:51.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 -- 172.21.15.129:0/3558801670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 msgr2=0x7f45041540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:51.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 --2- 172.21.15.129:0/3558801670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041540a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009df0 tx=0x7f44fc02bf90 comp rx=0 tx=0).stop 2024-03-23T15:57:51.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 -- 172.21.15.129:0/3558801670 shutdown_connections 2024-03-23T15:57:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 --2- 172.21.15.129:0/3558801670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041540a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 -- 172.21.15.129:0/3558801670 >> 172.21.15.129:0/3558801670 conn(0x7f4504071300 msgr2=0x7f45040716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 -- 172.21.15.129:0/3558801670 shutdown_connections 2024-03-23T15:57:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.770+0000 7f450b15c640 1 -- 172.21.15.129:0/3558801670 wait complete. 2024-03-23T15:57:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.771+0000 7f450b15c640 1 Processor -- start 2024-03-23T15:57:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.771+0000 7f450b15c640 1 -- start start 2024-03-23T15:57:51.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.772+0000 7f450b15c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f450406cf70 0x7f4504159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:51.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.772+0000 7f450b15c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:51.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.772+0000 7f450b15c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f450415dba0 0x7f45041f0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:51.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.772+0000 7f450b15c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f450414b710 con 0x7f450406cf70 2024-03-23T15:57:51.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.772+0000 7f450b15c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4504158880 con 0x7f450415dba0 2024-03-23T15:57:51.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.772+0000 7f450b15c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f450415aed0 con 0x7f45040cca30 2024-03-23T15:57:51.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.773+0000 7f45096f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f450415dba0 0x7f45041f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:51.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.773+0000 7f4508ef8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f450406cf70 0x7f4504159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:51.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.773+0000 7f4503fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:51.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.773+0000 7f45096f9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f450415dba0 0x7f45041f0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:58490/0 (socket says 172.21.15.129:58490) 2024-03-23T15:57:51.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.773+0000 7f45096f9640 1 -- 172.21.15.129:0/2527403082 learned_addr learned my addr 172.21.15.129:0/2527403082 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:51.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.776+0000 7f4508ef8640 1 -- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 msgr2=0x7f45041f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:51.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.776+0000 7f4508ef8640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:51.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.776+0000 7f4508ef8640 1 -- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f450415dba0 msgr2=0x7f45041f0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:51.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.776+0000 7f4508ef8640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f450415dba0 0x7f45041f0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:51.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.776+0000 7f4508ef8640 1 -- 172.21.15.129:0/2527403082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc009a70 con 0x7f450406cf70 2024-03-23T15:57:51.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.779+0000 7f4508ef8640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f450406cf70 0x7f4504159430 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f44f800c3c0 tx=0x7f44f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:51.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.781+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80052d0 con 0x7f450406cf70 2024-03-23T15:57:51.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.781+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450415b150 con 0x7f450406cf70 2024-03-23T15:57:51.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.781+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45041582c0 con 0x7f450406cf70 2024-03-23T15:57:51.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.782+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45040686f0 con 0x7f450406cf70 2024-03-23T15:57:51.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.783+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f8007df0 con 0x7f450406cf70 2024-03-23T15:57:51.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:51.783+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80035f0 con 0x7f450406cf70 2024-03-23T15:57:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.036+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f44f8003750 con 0x7f450406cf70 2024-03-23T15:57:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.037+0000 7f45017fa640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44e806b810 0x7f44e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.037+0000 7f4503fff640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44e806b810 0x7f44e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.038+0000 7f4503fff640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44e806b810 0x7f44e806dcd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f45040ce2e0 tx=0x7f44f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:52.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.050+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f44f80582b0 con 0x7f450406cf70 2024-03-23T15:57:52.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.050+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44f8054550 con 0x7f450406cf70 2024-03-23T15:57:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.291+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4504075740 con 0x7f450406cf70 2024-03-23T15:57:52.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.306+0000 7f45017fa640 1 -- 172.21.15.129:0/2527403082 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f44f8021090 con 0x7f450406cf70 2024-03-23T15:57:52.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:52.309 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:52.309 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:52.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44e806b810 msgr2=0x7f44e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:52.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44e806b810 0x7f44e806dcd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f45040ce2e0 tx=0x7f44f0009040 comp rx=0 tx=0).stop 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f450406cf70 msgr2=0x7f4504159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f450406cf70 0x7f4504159430 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f44f800c3c0 tx=0x7f44f800c880 comp rx=0 tx=0).stop 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 shutdown_connections 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f44e806b810 0x7f44e806dcd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f450415dba0 0x7f45041f0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f45040cca30 0x7f45041f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 --2- 172.21.15.129:0/2527403082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f450406cf70 0x7f4504159430 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 >> 172.21.15.129:0/2527403082 conn(0x7f4504071300 msgr2=0x7f45040716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 shutdown_connections 2024-03-23T15:57:52.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:52.309+0000 7f450b15c640 1 -- 172.21.15.129:0/2527403082 wait complete. 2024-03-23T15:57:52.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:52.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:53.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:54.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.029+0000 7f88ac1c7640 1 Processor -- start 2024-03-23T15:57:54.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.029+0000 7f88ac1c7640 1 -- start start 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.030+0000 7f88ac1c7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f88a415c000 0x7f88a415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.030+0000 7f88ac1c7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a415b290 con 0x7f88a40ccd60 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.030+0000 7f88ac1c7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a40ce800 con 0x7f88a40ceb00 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.030+0000 7f88ac1c7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f88a4154a70 con 0x7f88a415c000 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.031+0000 7f88a9762640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f88a40ceb00 0x7f88a415b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:52928/0 (socket says 172.21.15.129:52928) 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.031+0000 7f88a9762640 1 -- 172.21.15.129:0/4164054942 learned_addr learned my addr 172.21.15.129:0/4164054942 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.031+0000 7f88aa764640 1 --2- 172.21.15.129:0/4164054942 >> v2:172.21.15.129:3300/0 conn(0x7f88a415c000 0x7f88a415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:54.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.034+0000 7f88a8f61640 1 -- 172.21.15.129:0/4164054942 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3496365839 0 0) 0x7f88a40ce800 con 0x7f88a40ceb00 2024-03-23T15:57:54.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.034+0000 7f88a8f61640 1 -- 172.21.15.129:0/4164054942 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8890003520 con 0x7f88a40ceb00 2024-03-23T15:57:54.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.034+0000 7f88aa764640 1 -- 172.21.15.129:0/4164054942 >> v1:172.21.15.157:6789/0 conn(0x7f88a40ceb00 legacy=0x7f88a415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.034+0000 7f88aa764640 1 -- 172.21.15.129:0/4164054942 >> v1:172.21.15.129:6789/0 conn(0x7f88a40ccd60 legacy=0x7f88a406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.034+0000 7f88aa764640 1 -- 172.21.15.129:0/4164054942 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a40cd950 con 0x7f88a415c000 2024-03-23T15:57:54.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.038+0000 7f88aa764640 1 --2- 172.21.15.129:0/4164054942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a415c000 0x7f88a415ad50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f889c009d90 tx=0x7f889c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64bd3770f212893e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:54.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.043+0000 7f88a8f61640 1 -- 172.21.15.129:0/4164054942 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c02cd80 con 0x7f88a415c000 2024-03-23T15:57:54.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.043+0000 7f88a8f61640 1 -- 172.21.15.129:0/4164054942 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889c034440 con 0x7f88a415c000 2024-03-23T15:57:54.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.043+0000 7f88a8f61640 1 -- 172.21.15.129:0/4164054942 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c0347d0 con 0x7f88a415c000 2024-03-23T15:57:54.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 -- 172.21.15.129:0/4164054942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a415c000 msgr2=0x7f88a415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/4164054942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a415c000 0x7f88a415ad50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f889c009d90 tx=0x7f889c02bfd0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 -- 172.21.15.129:0/4164054942 shutdown_connections 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/4164054942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a415c000 0x7f88a415ad50 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 -- 172.21.15.129:0/4164054942 >> 172.21.15.129:0/4164054942 conn(0x7f88a40b1840 msgr2=0x7f88a40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 -- 172.21.15.129:0/4164054942 shutdown_connections 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.044+0000 7f88ac1c7640 1 -- 172.21.15.129:0/4164054942 wait complete. 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.045+0000 7f88ac1c7640 1 Processor -- start 2024-03-23T15:57:54.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.045+0000 7f88ac1c7640 1 -- start start 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88ac1c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88a40ccd60 0x7f88a41e46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88ac1c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 0x7f88a414e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88ac1c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88a415c000 0x7f88a414ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88ac1c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f88a41e5c30 con 0x7f88a40ceb00 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88ac1c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f88a41f46d0 con 0x7f88a415c000 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88ac1c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f88a41f4c10 con 0x7f88a40ccd60 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.046+0000 7f88a9762640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 0x7f88a414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:54.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.047+0000 7f88aa764640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88a415c000 0x7f88a414ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:54.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.047+0000 7f88a9f63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88a40ccd60 0x7f88a41e46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:54.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.047+0000 7f88a9762640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 0x7f88a414e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57558/0 (socket says 172.21.15.129:57558) 2024-03-23T15:57:54.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.047+0000 7f88a9762640 1 -- 172.21.15.129:0/2191435198 learned_addr learned my addr 172.21.15.129:0/2191435198 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:54.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.050+0000 7f88a9762640 1 -- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88a40ccd60 msgr2=0x7f88a41e46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.050+0000 7f88a9762640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88a40ccd60 0x7f88a41e46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.050+0000 7f88a9762640 1 -- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88a415c000 msgr2=0x7f88a414ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.050+0000 7f88a9762640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88a415c000 0x7f88a414ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.050+0000 7f88a9762640 1 -- 172.21.15.129:0/2191435198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c009a70 con 0x7f88a40ceb00 2024-03-23T15:57:54.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.053+0000 7f88a9762640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 0x7f88a414e160 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f889400c430 tx=0x7f889400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:54.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.057+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894005ce0 con 0x7f88a40ceb00 2024-03-23T15:57:54.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.057+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894005e40 con 0x7f88a40ceb00 2024-03-23T15:57:54.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.057+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a41e3990 con 0x7f88a40ceb00 2024-03-23T15:57:54.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.057+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88940072f0 con 0x7f88a40ceb00 2024-03-23T15:57:54.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.057+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a41f4930 con 0x7f88a40ceb00 2024-03-23T15:57:54.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.058+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a404f530 con 0x7f88a40ceb00 2024-03-23T15:57:54.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.082+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8894007450 con 0x7f88a40ceb00 2024-03-23T15:57:54.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.083+0000 7f889a7fc640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f888806b860 0x7f888806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:54.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.083+0000 7f88a9f63640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f888806b860 0x7f888806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:54.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.083+0000 7f88a9f63640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f888806b860 0x7f888806dd20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f88a00016e0 tx=0x7f88a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:54.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.085+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8894058480 con 0x7f88a40ceb00 2024-03-23T15:57:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.352+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8894054720 con 0x7f88a40ceb00 2024-03-23T15:57:54.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.586+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88a40b5790 con 0x7f88a40ceb00 2024-03-23T15:57:54.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.598+0000 7f889a7fc640 1 -- 172.21.15.129:0/2191435198 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8894017020 con 0x7f88a40ceb00 2024-03-23T15:57:54.600 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:54.600 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:54.601 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:54.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f888806b860 msgr2=0x7f888806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f888806b860 0x7f888806dd20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f88a00016e0 tx=0x7f88a0009040 comp rx=0 tx=0).stop 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 msgr2=0x7f88a414e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 0x7f88a414e160 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f889400c430 tx=0x7f889400c8f0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 shutdown_connections 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f888806b860 0x7f888806dd20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f88a415c000 0x7f88a414ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88a40ceb00 0x7f88a414e160 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 --2- 172.21.15.129:0/2191435198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f88a40ccd60 0x7f88a41e46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 >> 172.21.15.129:0/2191435198 conn(0x7f88a40b1840 msgr2=0x7f88a40b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 shutdown_connections 2024-03-23T15:57:54.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:54.601+0000 7f88ac1c7640 1 -- 172.21.15.129:0/2191435198 wait complete. 2024-03-23T15:57:55.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:55.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:56.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:56.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.319+0000 7f97b74df640 1 Processor -- start 2024-03-23T15:57:56.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.319+0000 7f97b74df640 1 -- start start 2024-03-23T15:57:56.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.320+0000 7f97b74df640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f97b015d9c0 0x7f97b006c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:56.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.320+0000 7f97b74df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b00ca4a0 0x7f97b00cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:56.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.320+0000 7f97b74df640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b00cb810 con 0x7f97b015dda0 2024-03-23T15:57:56.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.320+0000 7f97b74df640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f97b00cac00 con 0x7f97b015d9c0 2024-03-23T15:57:56.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.320+0000 7f97b74df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97b00caee0 con 0x7f97b00ca4a0 2024-03-23T15:57:56.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.320+0000 7f97b4a7a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b00ca4a0 0x7f97b00cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:56.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.321+0000 7f97b527b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f97b015d9c0 0x7f97b006c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:56.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.321+0000 7f97b5a7c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f97b015dda0 0x7f97b015a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:52942/0 (socket says 172.21.15.129:52942) 2024-03-23T15:57:56.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.321+0000 7f97b5a7c640 1 -- 172.21.15.129:0/838710471 learned_addr learned my addr 172.21.15.129:0/838710471 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:56.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.323+0000 7f97b527b640 1 -- 172.21.15.129:0/838710471 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b00ca4a0 msgr2=0x7f97b00cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:57:56.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.323+0000 7f97b527b640 1 --2- 172.21.15.129:0/838710471 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b00ca4a0 0x7f97b00cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.323+0000 7f97b527b640 1 -- 172.21.15.129:0/838710471 >> v1:172.21.15.157:6789/0 conn(0x7f97b015dda0 legacy=0x7f97b015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:56.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.323+0000 7f97b527b640 1 -- 172.21.15.129:0/838710471 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b00cb1c0 con 0x7f97b015d9c0 2024-03-23T15:57:56.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.325+0000 7f97b527b640 1 --2- 172.21.15.129:0/838710471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b015d9c0 0x7f97b006c5a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0097a0 tx=0x7f97ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56f6badb65295b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:56.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97a7fff640 1 -- 172.21.15.129:0/838710471 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac031830 con 0x7f97b015d9c0 2024-03-23T15:57:56.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97a7fff640 1 -- 172.21.15.129:0/838710471 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac031990 con 0x7f97b015d9c0 2024-03-23T15:57:56.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97a7fff640 1 -- 172.21.15.129:0/838710471 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac031d20 con 0x7f97b015d9c0 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 -- 172.21.15.129:0/838710471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b015d9c0 msgr2=0x7f97b006c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 --2- 172.21.15.129:0/838710471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b015d9c0 0x7f97b006c5a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0097a0 tx=0x7f97ac02b8d0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 -- 172.21.15.129:0/838710471 shutdown_connections 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 --2- 172.21.15.129:0/838710471 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b00ca4a0 0x7f97b00cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 --2- 172.21.15.129:0/838710471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b015d9c0 0x7f97b006c5a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 -- 172.21.15.129:0/838710471 >> 172.21.15.129:0/838710471 conn(0x7f97b00cfb20 msgr2=0x7f97b00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.329+0000 7f97b74df640 1 -- 172.21.15.129:0/838710471 shutdown_connections 2024-03-23T15:57:56.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.330+0000 7f97b74df640 1 -- 172.21.15.129:0/838710471 wait complete. 2024-03-23T15:57:56.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.331+0000 7f97b74df640 1 Processor -- start 2024-03-23T15:57:56.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.331+0000 7f97b74df640 1 -- start start 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b74df640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 0x7f97b01f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b74df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b015d9c0 0x7f97b0157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b74df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b015dda0 0x7f97b01f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b74df640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01f4180 con 0x7f97b00ca4a0 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b74df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f97b015af70 con 0x7f97b015d9c0 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b74df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97b015b250 con 0x7f97b015dda0 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b527b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 0x7f97b01f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b5a7c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b015dda0 0x7f97b01f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b4a7a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b015d9c0 0x7f97b0157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b527b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 0x7f97b01f08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57568/0 (socket says 172.21.15.129:57568) 2024-03-23T15:57:56.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.332+0000 7f97b527b640 1 -- 172.21.15.129:0/4107487147 learned_addr learned my addr 172.21.15.129:0/4107487147 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:56.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.336+0000 7f97b527b640 1 -- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b015dda0 msgr2=0x7f97b01f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:56.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.336+0000 7f97b527b640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b015dda0 0x7f97b01f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.336+0000 7f97b527b640 1 -- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b015d9c0 msgr2=0x7f97b0157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:56.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.336+0000 7f97b527b640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b015d9c0 0x7f97b0157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.336+0000 7f97b527b640 1 -- 172.21.15.129:0/4107487147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac009400 con 0x7f97b00ca4a0 2024-03-23T15:57:56.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.339+0000 7f97b527b640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 0x7f97b01f08b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0098c0 tx=0x7f97ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:56.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.342+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac03a6f0 con 0x7f97b00ca4a0 2024-03-23T15:57:56.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.342+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac03a870 con 0x7f97b00ca4a0 2024-03-23T15:57:56.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.342+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac03ac20 con 0x7f97b00ca4a0 2024-03-23T15:57:56.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.342+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b015b530 con 0x7f97b00ca4a0 2024-03-23T15:57:56.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.343+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b01f00b0 con 0x7f97b00ca4a0 2024-03-23T15:57:56.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.344+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b00cf4c0 con 0x7f97b00ca4a0 2024-03-23T15:57:56.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.365+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97ac04c3b0 con 0x7f97b00ca4a0 2024-03-23T15:57:56.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.366+0000 7f97a5ffb640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f979406b860 0x7f979406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:56.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.366+0000 7f97b4a7a640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f979406b860 0x7f979406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:56.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.366+0000 7f97b4a7a640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f979406b860 0x7f979406dd20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f97b01e7320 tx=0x7f97a0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:56.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.369+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f97ac07ab60 con 0x7f97b00ca4a0 2024-03-23T15:57:56.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.624+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97ac076e00 con 0x7f97b00ca4a0 2024-03-23T15:57:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.853+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97b0075740 con 0x7f97b00ca4a0 2024-03-23T15:57:56.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.867+0000 7f97a5ffb640 1 -- 172.21.15.129:0/4107487147 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f97ac07a4c0 con 0x7f97b00ca4a0 2024-03-23T15:57:56.869 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:56.869 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:56.870 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f979406b860 msgr2=0x7f979406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f979406b860 0x7f979406dd20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f97b01e7320 tx=0x7f97a0009190 comp rx=0 tx=0).stop 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 msgr2=0x7f97b01f08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 0x7f97b01f08b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0098c0 tx=0x7f97ac0046c0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 shutdown_connections 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b015dda0 0x7f97b01f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f979406b860 0x7f979406dd20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b015d9c0 0x7f97b0157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 --2- 172.21.15.129:0/4107487147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97b00ca4a0 0x7f97b01f08b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 >> 172.21.15.129:0/4107487147 conn(0x7f97b00cfb20 msgr2=0x7f97b0156cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 shutdown_connections 2024-03-23T15:57:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:56.869+0000 7f97b74df640 1 -- 172.21.15.129:0/4107487147 wait complete. 2024-03-23T15:57:57.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:57.487 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:57:58.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:57:58.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.591+0000 7f2da3f43640 1 Processor -- start 2024-03-23T15:57:58.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.591+0000 7f2da3f43640 1 -- start start 2024-03-23T15:57:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.592+0000 7f2da3f43640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d9c1544b0 0x7f2d9c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.593+0000 7f2da3f43640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9c0cbcb0 con 0x7f2d9c0c6000 2024-03-23T15:57:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.593+0000 7f2da3f43640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9c15b280 con 0x7f2d9c0caa00 2024-03-23T15:57:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.593+0000 7f2da3f43640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9c0c5be0 con 0x7f2d9c1544b0 2024-03-23T15:57:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.593+0000 7f2da24e0640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f2d9c0c6000 0x7f2d9c164c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48510/0 (socket says 172.21.15.129:48510) 2024-03-23T15:57:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.593+0000 7f2da24e0640 1 -- 172.21.15.129:0/1028022802 learned_addr learned my addr 172.21.15.129:0/1028022802 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:57:58.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.594+0000 7f2da1cdf640 1 --2- 172.21.15.129:0/1028022802 >> v2:172.21.15.157:3300/0 conn(0x7f2d9c1544b0 0x7f2d9c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:58.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.596+0000 7f2da0cdd640 1 -- 172.21.15.129:0/1028022802 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824426404 0 0) 0x7f2d9c0cbcb0 con 0x7f2d9c0c6000 2024-03-23T15:57:58.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.596+0000 7f2da0cdd640 1 -- 172.21.15.129:0/1028022802 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d88003520 con 0x7f2d9c0c6000 2024-03-23T15:57:58.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.598+0000 7f2da1cdf640 1 -- 172.21.15.129:0/1028022802 >> v1:172.21.15.157:6789/0 conn(0x7f2d9c0caa00 legacy=0x7f2d9c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:58.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.598+0000 7f2da1cdf640 1 -- 172.21.15.129:0/1028022802 >> v1:172.21.15.129:6789/0 conn(0x7f2d9c0c6000 legacy=0x7f2d9c164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:58.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.598+0000 7f2da1cdf640 1 -- 172.21.15.129:0/1028022802 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c15a9e0 con 0x7f2d9c1544b0 2024-03-23T15:57:58.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.605+0000 7f2da1cdf640 1 --2- 172.21.15.129:0/1028022802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c15a2e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097a0 tx=0x7f2d9802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f51357ccbd8915c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.610+0000 7f2da0cdd640 1 -- 172.21.15.129:0/1028022802 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9802f810 con 0x7f2d9c1544b0 2024-03-23T15:57:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.610+0000 7f2da0cdd640 1 -- 172.21.15.129:0/1028022802 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9802f970 con 0x7f2d9c1544b0 2024-03-23T15:57:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.610+0000 7f2da0cdd640 1 -- 172.21.15.129:0/1028022802 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9802fd00 con 0x7f2d9c1544b0 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 -- 172.21.15.129:0/1028022802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c1544b0 msgr2=0x7f2d9c15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 --2- 172.21.15.129:0/1028022802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c15a2e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097a0 tx=0x7f2d9802b9f0 comp rx=0 tx=0).stop 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 -- 172.21.15.129:0/1028022802 shutdown_connections 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 --2- 172.21.15.129:0/1028022802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c15a2e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 -- 172.21.15.129:0/1028022802 >> 172.21.15.129:0/1028022802 conn(0x7f2d9c061b50 msgr2=0x7f2d9c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 -- 172.21.15.129:0/1028022802 shutdown_connections 2024-03-23T15:57:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.611+0000 7f2da3f43640 1 -- 172.21.15.129:0/1028022802 wait complete. 2024-03-23T15:57:58.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.612+0000 7f2da3f43640 1 Processor -- start 2024-03-23T15:57:58.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.612+0000 7f2da3f43640 1 -- start start 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da3f43640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d9c0c6000 0x7f2d9c0cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da3f43640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c0caa00 0x7f2d9c147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da3f43640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da3f43640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c0d0880 con 0x7f2d9c1544b0 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da3f43640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c0cfcd0 con 0x7f2d9c0caa00 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da3f43640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9c1550a0 con 0x7f2d9c0c6000 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da24e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da14de640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c0caa00 0x7f2d9c147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.613+0000 7f2da1cdf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d9c0c6000 0x7f2d9c0cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.614+0000 7f2da24e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c147a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57592/0 (socket says 172.21.15.129:57592) 2024-03-23T15:57:58.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.614+0000 7f2da24e0640 1 -- 172.21.15.129:0/514995780 learned_addr learned my addr 172.21.15.129:0/514995780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:57:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.617+0000 7f2da24e0640 1 -- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d9c0c6000 msgr2=0x7f2d9c0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.617+0000 7f2da24e0640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d9c0c6000 0x7f2d9c0cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.617+0000 7f2da24e0640 1 -- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c0caa00 msgr2=0x7f2d9c147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.617+0000 7f2da24e0640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c0caa00 0x7f2d9c147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.617+0000 7f2da24e0640 1 -- 172.21.15.129:0/514995780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d98009400 con 0x7f2d9c1544b0 2024-03-23T15:57:58.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.619+0000 7f2da24e0640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c147a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400cb40 tx=0x7f2d9400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.624+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400aca0 con 0x7f2d9c1544b0 2024-03-23T15:57:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.624+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9400ae00 con 0x7f2d9c1544b0 2024-03-23T15:57:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.624+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c0cfee0 con 0x7f2d9c1544b0 2024-03-23T15:57:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.624+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94012620 con 0x7f2d9c1544b0 2024-03-23T15:57:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.624+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c0d04a0 con 0x7f2d9c1544b0 2024-03-23T15:57:58.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.625+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d9c075740 con 0x7f2d9c1544b0 2024-03-23T15:57:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.647+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2d94007070 con 0x7f2d9c1544b0 2024-03-23T15:57:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.648+0000 7f2d927fc640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2d8006b840 0x7f2d8006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:57:58.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.648+0000 7f2da1cdf640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2d8006b840 0x7f2d8006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:57:58.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.649+0000 7f2da1cdf640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2d8006b840 0x7f2d8006dd00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2d98036000 tx=0x7f2d98008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:57:58.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.918+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2d94058df0 con 0x7f2d9c1544b0 2024-03-23T15:57:58.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:58.918+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2d94055090 con 0x7f2d9c1544b0 2024-03-23T15:57:59.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.172+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d9c14e150 con 0x7f2d9c1544b0 2024-03-23T15:57:59.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.199+0000 7f2d927fc640 1 -- 172.21.15.129:0/514995780 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2d94058750 con 0x7f2d9c1544b0 2024-03-23T15:57:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:57:59.202 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:57:59.202 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:57:59.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2d8006b840 msgr2=0x7f2d8006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:59.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2d8006b840 0x7f2d8006dd00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2d98036000 tx=0x7f2d98008030 comp rx=0 tx=0).stop 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 msgr2=0x7f2d9c147a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c147a90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400cb40 tx=0x7f2d9400a060 comp rx=0 tx=0).stop 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 shutdown_connections 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9c1544b0 0x7f2d9c147a90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2d8006b840 0x7f2d8006dd00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9c0caa00 0x7f2d9c147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 --2- 172.21.15.129:0/514995780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d9c0c6000 0x7f2d9c0cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 >> 172.21.15.129:0/514995780 conn(0x7f2d9c061b50 msgr2=0x7f2d9c158130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 shutdown_connections 2024-03-23T15:57:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:57:59.202+0000 7f2da3f43640 1 -- 172.21.15.129:0/514995780 wait complete. 2024-03-23T15:57:59.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:57:59.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:00.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:00.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.920+0000 7fd4649e2640 1 Processor -- start 2024-03-23T15:58:00.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.920+0000 7fd4649e2640 1 -- start start 2024-03-23T15:58:00.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.921+0000 7fd4649e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601544d0 0x7fd4600c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:00.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.921+0000 7fd4649e2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd46015e470 0x7fd46015ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:00.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.921+0000 7fd4649e2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd46015e8b0 con 0x7fd46006b930 2024-03-23T15:58:00.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.921+0000 7fd4649e2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd460072a70 con 0x7fd46015e470 2024-03-23T15:58:00.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.921+0000 7fd4649e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd46015b2c0 con 0x7fd4601544d0 2024-03-23T15:58:00.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.922+0000 7fd45e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601544d0 0x7fd4600c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:00.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.922+0000 7fd45dd9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd46015e470 0x7fd46015ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:00.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.922+0000 7fd45ed9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd46006b930 0x7fd4600ccb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:59392/0 (socket says 172.21.15.129:59392) 2024-03-23T15:58:00.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.922+0000 7fd45ed9d640 1 -- 172.21.15.129:0/3166339086 learned_addr learned my addr 172.21.15.129:0/3166339086 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.925+0000 7fd45e59c640 1 -- 172.21.15.129:0/3166339086 >> v2:172.21.15.157:3300/0 conn(0x7fd46015e470 msgr2=0x7fd46015ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.925+0000 7fd45e59c640 1 --2- 172.21.15.129:0/3166339086 >> v2:172.21.15.157:3300/0 conn(0x7fd46015e470 0x7fd46015ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.925+0000 7fd45e59c640 1 -- 172.21.15.129:0/3166339086 >> v1:172.21.15.157:6789/0 conn(0x7fd46006b930 legacy=0x7fd4600ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.925+0000 7fd45e59c640 1 -- 172.21.15.129:0/3166339086 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd460072ce0 con 0x7fd4601544d0 2024-03-23T15:58:00.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.927+0000 7fd45e59c640 1 --2- 172.21.15.129:0/3166339086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601544d0 0x7fd4600c7ca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd4500097a0 tx=0x7fd45002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2b666a6ff44ec5f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.932+0000 7fd45d59a640 1 -- 172.21.15.129:0/3166339086 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450031830 con 0x7fd4601544d0 2024-03-23T15:58:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.932+0000 7fd45d59a640 1 -- 172.21.15.129:0/3166339086 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd450031990 con 0x7fd4601544d0 2024-03-23T15:58:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.932+0000 7fd45d59a640 1 -- 172.21.15.129:0/3166339086 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450031d20 con 0x7fd4601544d0 2024-03-23T15:58:00.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 -- 172.21.15.129:0/3166339086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601544d0 msgr2=0x7fd4600c7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:00.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 --2- 172.21.15.129:0/3166339086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601544d0 0x7fd4600c7ca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd4500097a0 tx=0x7fd45002b8d0 comp rx=0 tx=0).stop 2024-03-23T15:58:00.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 -- 172.21.15.129:0/3166339086 shutdown_connections 2024-03-23T15:58:00.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 --2- 172.21.15.129:0/3166339086 >> v2:172.21.15.157:3300/0 conn(0x7fd46015e470 0x7fd46015ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:00.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 --2- 172.21.15.129:0/3166339086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601544d0 0x7fd4600c7ca0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 -- 172.21.15.129:0/3166339086 >> 172.21.15.129:0/3166339086 conn(0x7fd46006e230 msgr2=0x7fd46006e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 -- 172.21.15.129:0/3166339086 shutdown_connections 2024-03-23T15:58:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.933+0000 7fd4649e2640 1 -- 172.21.15.129:0/3166339086 wait complete. 2024-03-23T15:58:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.934+0000 7fd4649e2640 1 Processor -- start 2024-03-23T15:58:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.934+0000 7fd4649e2640 1 -- start start 2024-03-23T15:58:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd4649e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd46006b930 0x7fd4601f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd4649e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4601544d0 0x7fd4600cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd4649e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd46015e470 0x7fd4601f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd4649e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd46014eb40 con 0x7fd4601544d0 2024-03-23T15:58:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd4649e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601f1070 con 0x7fd46006b930 2024-03-23T15:58:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd4649e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd46014e600 con 0x7fd46015e470 2024-03-23T15:58:00.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.935+0000 7fd45e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd46006b930 0x7fd4601f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:00.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.936+0000 7fd45ed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd46015e470 0x7fd4601f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:00.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.936+0000 7fd45dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4601544d0 0x7fd4600cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:00.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.936+0000 7fd45e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd46006b930 0x7fd4601f0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:51716/0 (socket says 172.21.15.129:51716) 2024-03-23T15:58:00.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.936+0000 7fd45e59c640 1 -- 172.21.15.129:0/452935323 learned_addr learned my addr 172.21.15.129:0/452935323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:00.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.939+0000 7fd45dd9b640 1 -- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd46015e470 msgr2=0x7fd4601f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:00.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.939+0000 7fd45dd9b640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd46015e470 0x7fd4601f16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:00.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.939+0000 7fd45dd9b640 1 -- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd46006b930 msgr2=0x7fd4601f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:00.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.939+0000 7fd45dd9b640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd46006b930 0x7fd4601f0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:00.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.939+0000 7fd45dd9b640 1 -- 172.21.15.129:0/452935323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd450009400 con 0x7fd4601544d0 2024-03-23T15:58:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.942+0000 7fd45dd9b640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4601544d0 0x7fd4600cbdc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd45400d620 tx=0x7fd45400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.945+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4540125f0 con 0x7fd4601544d0 2024-03-23T15:58:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.945+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46014d770 con 0x7fd4601544d0 2024-03-23T15:58:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.945+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46014dcd0 con 0x7fd4601544d0 2024-03-23T15:58:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.946+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd460075740 con 0x7fd4601544d0 2024-03-23T15:58:00.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.971+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd454012750 con 0x7fd4601544d0 2024-03-23T15:58:00.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.971+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454012ae0 con 0x7fd4601544d0 2024-03-23T15:58:00.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.972+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd454012cc0 con 0x7fd4601544d0 2024-03-23T15:58:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.972+0000 7fd44effd640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd44006b810 0x7fd44006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.973+0000 7fd45e59c640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd44006b810 0x7fd44006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.973+0000 7fd45e59c640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd44006b810 0x7fd44006dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd45002b910 tx=0x7fd45003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:00.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:00.976+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fd4540598d0 con 0x7fd4601544d0 2024-03-23T15:58:01.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.239+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd454055b70 con 0x7fd4601544d0 2024-03-23T15:58:01.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.462+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4600c3d50 con 0x7fd4601544d0 2024-03-23T15:58:01.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.476+0000 7fd44effd640 1 -- 172.21.15.129:0/452935323 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fd454059230 con 0x7fd4601544d0 2024-03-23T15:58:01.479 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:01.479 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:01.479 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd44006b810 msgr2=0x7fd44006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd44006b810 0x7fd44006dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd45002b910 tx=0x7fd45003c040 comp rx=0 tx=0).stop 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4601544d0 msgr2=0x7fd4600cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4601544d0 0x7fd4600cbdc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd45400d620 tx=0x7fd45400dae0 comp rx=0 tx=0).stop 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 shutdown_connections 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd46015e470 0x7fd4601f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4601544d0 0x7fd4600cbdc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fd44006b810 0x7fd44006dcd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 --2- 172.21.15.129:0/452935323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd46006b930 0x7fd4601f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 >> 172.21.15.129:0/452935323 conn(0x7fd46006e230 msgr2=0x7fd46015b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 shutdown_connections 2024-03-23T15:58:01.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:01.478+0000 7fd4649e2640 1 -- 172.21.15.129:0/452935323 wait complete. 2024-03-23T15:58:02.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:02.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:03.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:03.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.197+0000 7f09f5084640 1 Processor -- start 2024-03-23T15:58:03.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.197+0000 7f09f5084640 1 -- start start 2024-03-23T15:58:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.198+0000 7f09f5084640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f09f01544b0 0x7f09f015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.198+0000 7f09f5084640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f00cbcb0 con 0x7f09f00caa00 2024-03-23T15:58:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.198+0000 7f09f5084640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f015b280 con 0x7f09f00c6000 2024-03-23T15:58:03.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.198+0000 7f09f5084640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f09f00c5be0 con 0x7f09f01544b0 2024-03-23T15:58:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.199+0000 7f09e7fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f09f00caa00 0x7f09f006bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55184/0 (socket says 172.21.15.129:55184) 2024-03-23T15:58:03.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.199+0000 7f09e7fff640 1 -- 172.21.15.129:0/3277831696 learned_addr learned my addr 172.21.15.129:0/3277831696 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:03.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.200+0000 7f09eed9d640 1 --2- 172.21.15.129:0/3277831696 >> v2:172.21.15.157:3300/0 conn(0x7f09f01544b0 0x7f09f015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.202+0000 7f09ee59c640 1 -- 172.21.15.129:0/3277831696 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762067225 0 0) 0x7f09f00cbcb0 con 0x7f09f00caa00 2024-03-23T15:58:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.202+0000 7f09ee59c640 1 -- 172.21.15.129:0/3277831696 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09dc003520 con 0x7f09f00caa00 2024-03-23T15:58:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.206+0000 7f09ee59c640 1 -- 172.21.15.129:0/3277831696 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4072821788 0 0) 0x7f09dc003520 con 0x7f09f00caa00 2024-03-23T15:58:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.206+0000 7f09eed9d640 1 -- 172.21.15.129:0/3277831696 >> v1:172.21.15.157:6789/0 conn(0x7f09f00c6000 legacy=0x7f09f0164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.206+0000 7f09eed9d640 1 -- 172.21.15.129:0/3277831696 >> v1:172.21.15.129:6789/0 conn(0x7f09f00caa00 legacy=0x7f09f006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.206+0000 7f09eed9d640 1 -- 172.21.15.129:0/3277831696 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f015a9e0 con 0x7f09f01544b0 2024-03-23T15:58:03.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.210+0000 7f09eed9d640 1 --2- 172.21.15.129:0/3277831696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f01544b0 0x7f09f015a2e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097a0 tx=0x7f09e002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c455a27592e67058 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:03.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09ee59c640 1 -- 172.21.15.129:0/3277831696 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8001670 con 0x7f09f01544b0 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09ee59c640 1 -- 172.21.15.129:0/3277831696 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e0031590 con 0x7f09f01544b0 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09ee59c640 1 -- 172.21.15.129:0/3277831696 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0031920 con 0x7f09f01544b0 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09f5084640 1 -- 172.21.15.129:0/3277831696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f01544b0 msgr2=0x7f09f015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09f5084640 1 --2- 172.21.15.129:0/3277831696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f01544b0 0x7f09f015a2e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097a0 tx=0x7f09e002b8b0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09f5084640 1 -- 172.21.15.129:0/3277831696 shutdown_connections 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09f5084640 1 --2- 172.21.15.129:0/3277831696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f01544b0 0x7f09f015a2e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.215+0000 7f09f5084640 1 -- 172.21.15.129:0/3277831696 >> 172.21.15.129:0/3277831696 conn(0x7f09f0061b50 msgr2=0x7f09f0061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.216+0000 7f09f5084640 1 -- 172.21.15.129:0/3277831696 shutdown_connections 2024-03-23T15:58:03.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.216+0000 7f09f5084640 1 -- 172.21.15.129:0/3277831696 wait complete. 2024-03-23T15:58:03.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.217+0000 7f09f5084640 1 Processor -- start 2024-03-23T15:58:03.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.217+0000 7f09f5084640 1 -- start start 2024-03-23T15:58:03.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09f5084640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09f00c6000 0x7f09f00cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:03.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09f5084640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f00caa00 0x7f09f01f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:03.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09f5084640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 0x7f09f006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:03.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09f5084640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09f0157ba0 con 0x7f09f01544b0 2024-03-23T15:58:03.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09f5084640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00cf170 con 0x7f09f00caa00 2024-03-23T15:58:03.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09f5084640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f09f0157320 con 0x7f09f00c6000 2024-03-23T15:58:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09ef59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 0x7f09f006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.218+0000 7f09e7fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f00caa00 0x7f09f01f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.219+0000 7f09eed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09f00c6000 0x7f09f00cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.219+0000 7f09ef59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 0x7f09f006bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38682/0 (socket says 172.21.15.129:38682) 2024-03-23T15:58:03.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.219+0000 7f09ef59e640 1 -- 172.21.15.129:0/1004882661 learned_addr learned my addr 172.21.15.129:0/1004882661 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:03.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.222+0000 7f09ef59e640 1 -- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09f00c6000 msgr2=0x7f09f00cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.222+0000 7f09ef59e640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09f00c6000 0x7f09f00cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.222+0000 7f09ef59e640 1 -- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f00caa00 msgr2=0x7f09f01f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.222+0000 7f09ef59e640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f00caa00 0x7f09f01f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.222+0000 7f09ef59e640 1 -- 172.21.15.129:0/1004882661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e0009400 con 0x7f09f01544b0 2024-03-23T15:58:03.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.224+0000 7f09ef59e640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 0x7f09f006bdc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c710 tx=0x7f09e800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.228+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8006d50 con 0x7f09f01544b0 2024-03-23T15:58:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.229+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e800ce20 con 0x7f09f01544b0 2024-03-23T15:58:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.229+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8003bb0 con 0x7f09f01544b0 2024-03-23T15:58:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.229+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f01f41a0 con 0x7f09f01544b0 2024-03-23T15:58:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.229+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f01578c0 con 0x7f09f01544b0 2024-03-23T15:58:03.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.230+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f0075740 con 0x7f09f01544b0 2024-03-23T15:58:03.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.493+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09e8003d10 con 0x7f09f01544b0 2024-03-23T15:58:03.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.493+0000 7f09e77fe640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09d006b840 0x7f09d006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:03.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.493+0000 7f09eed9d640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09d006b840 0x7f09d006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:03.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.494+0000 7f09eed9d640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09d006b840 0x7f09d006dd00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036000 tx=0x7f09e003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:03.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.510+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f09e8059250 con 0x7f09f01544b0 2024-03-23T15:58:03.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.510+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09e8059020 con 0x7f09f01544b0 2024-03-23T15:58:03.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.756+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09f00cb540 con 0x7f09f01544b0 2024-03-23T15:58:03.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.771+0000 7f09e77fe640 1 -- 172.21.15.129:0/1004882661 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f09e8058bb0 con 0x7f09f01544b0 2024-03-23T15:58:03.773 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:03.774 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:03.774 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.773+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09d006b840 msgr2=0x7f09d006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.773+0000 7f09f5084640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09d006b840 0x7f09d006dd00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036000 tx=0x7f09e003d040 comp rx=0 tx=0).stop 2024-03-23T15:58:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.773+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 msgr2=0x7f09f006bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.773+0000 7f09f5084640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 0x7f09f006bdc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c710 tx=0x7f09e800cbd0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 shutdown_connections 2024-03-23T15:58:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f01544b0 0x7f09f006bdc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f09d006b840 0x7f09d006dd00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f00caa00 0x7f09f01f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 --2- 172.21.15.129:0/1004882661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09f00c6000 0x7f09f00cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:03.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 >> 172.21.15.129:0/1004882661 conn(0x7f09f0061b50 msgr2=0x7f09f00b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:03.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 shutdown_connections 2024-03-23T15:58:03.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:03.774+0000 7f09f5084640 1 -- 172.21.15.129:0/1004882661 wait complete. 2024-03-23T15:58:04.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:04.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:05.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:05.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.492+0000 7fb0e107c640 1 Processor -- start 2024-03-23T15:58:05.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.492+0000 7fb0e107c640 1 -- start start 2024-03-23T15:58:05.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0e107c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc1544b0 0x7fb0dc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:05.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0e107c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb0dc0c6000 0x7fb0dc0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:05.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0e107c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0dc0cbcb0 con 0x7fb0dc0caa00 2024-03-23T15:58:05.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0e107c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb0dc0c5be0 con 0x7fb0dc0c6000 2024-03-23T15:58:05.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0e107c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb0dc0c69b0 con 0x7fb0dc1544b0 2024-03-23T15:58:05.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0da59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb0dc0caa00 0x7fb0dc06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55194/0 (socket says 172.21.15.129:55194) 2024-03-23T15:58:05.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0da59c640 1 -- 172.21.15.129:0/2321812798 learned_addr learned my addr 172.21.15.129:0/2321812798 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:05.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0dad9d640 1 --2- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc1544b0 0x7fb0dc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:05.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.493+0000 7fb0db59e640 1 --2- 172.21.15.129:0/2321812798 >> v2:172.21.15.157:3300/0 conn(0x7fb0dc0c6000 0x7fb0dc0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:05.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.496+0000 7fb0d9d9b640 1 -- 172.21.15.129:0/2321812798 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557152066 0 0) 0x7fb0dc0cbcb0 con 0x7fb0dc0caa00 2024-03-23T15:58:05.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.496+0000 7fb0d9d9b640 1 -- 172.21.15.129:0/2321812798 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c4003520 con 0x7fb0dc0caa00 2024-03-23T15:58:05.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.497+0000 7fb0db59e640 1 -- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc1544b0 msgr2=0x7fb0dc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:05.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.497+0000 7fb0db59e640 1 --2- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc1544b0 0x7fb0dc15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:05.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.497+0000 7fb0db59e640 1 -- 172.21.15.129:0/2321812798 >> v1:172.21.15.129:6789/0 conn(0x7fb0dc0caa00 legacy=0x7fb0dc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:05.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.497+0000 7fb0db59e640 1 -- 172.21.15.129:0/2321812798 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0dc15b280 con 0x7fb0dc0c6000 2024-03-23T15:58:05.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.500+0000 7fb0db59e640 1 --2- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc0c6470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009d90 tx=0x7fb0d402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69f7a3965a4d01f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:05.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0d9d9b640 1 -- 172.21.15.129:0/2321812798 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d402cce0 con 0x7fb0dc0c6000 2024-03-23T15:58:05.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0d9d9b640 1 -- 172.21.15.129:0/2321812798 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0d402ce40 con 0x7fb0dc0c6000 2024-03-23T15:58:05.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0d9d9b640 1 -- 172.21.15.129:0/2321812798 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4034660 con 0x7fb0dc0c6000 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 -- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc0c6000 msgr2=0x7fb0dc0c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 --2- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc0c6470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009d90 tx=0x7fb0d402bf30 comp rx=0 tx=0).stop 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 -- 172.21.15.129:0/2321812798 shutdown_connections 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 --2- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc0c6470 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 --2- 172.21.15.129:0/2321812798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc1544b0 0x7fb0dc15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 -- 172.21.15.129:0/2321812798 >> 172.21.15.129:0/2321812798 conn(0x7fb0dc061b50 msgr2=0x7fb0dc061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 -- 172.21.15.129:0/2321812798 shutdown_connections 2024-03-23T15:58:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.505+0000 7fb0e107c640 1 -- 172.21.15.129:0/2321812798 wait complete. 2024-03-23T15:58:05.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.507+0000 7fb0e107c640 1 Processor -- start 2024-03-23T15:58:05.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.507+0000 7fb0e107c640 1 -- start start 2024-03-23T15:58:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0e107c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc1f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0e107c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc0caa00 0x7fb0dc0cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0e107c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 0x7fb0dc0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0e107c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc0cda40 con 0x7fb0dc0c6000 2024-03-23T15:58:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0e107c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc0cdd20 con 0x7fb0dc1544b0 2024-03-23T15:58:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0e107c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb0dc157450 con 0x7fb0dc0caa00 2024-03-23T15:58:05.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0dad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc1f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:05.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0db59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 0x7fb0dc0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:05.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.508+0000 7fb0da59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc0caa00 0x7fb0dc0cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:05.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.509+0000 7fb0db59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 0x7fb0dc0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:51744/0 (socket says 172.21.15.129:51744) 2024-03-23T15:58:05.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.509+0000 7fb0db59e640 1 -- 172.21.15.129:0/1665823836 learned_addr learned my addr 172.21.15.129:0/1665823836 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:05.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.512+0000 7fb0db59e640 1 -- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc0caa00 msgr2=0x7fb0dc0cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:05.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.512+0000 7fb0db59e640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc0caa00 0x7fb0dc0cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:05.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.512+0000 7fb0db59e640 1 -- 172.21.15.129:0/1665823836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc0c6000 msgr2=0x7fb0dc1f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:05.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.512+0000 7fb0db59e640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc1f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:05.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.512+0000 7fb0db59e640 1 -- 172.21.15.129:0/1665823836 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d4009a70 con 0x7fb0dc1544b0 2024-03-23T15:58:05.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.516+0000 7fb0db59e640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 0x7fb0dc0cf660 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009eb0 tx=0x7fb0d4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.520+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d403d600 con 0x7fb0dc1544b0 2024-03-23T15:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.520+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0d403d780 con 0x7fb0dc1544b0 2024-03-23T15:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.520+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0dc1576c0 con 0x7fb0dc1544b0 2024-03-23T15:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.520+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d403db30 con 0x7fb0dc1544b0 2024-03-23T15:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.520+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0dc154bd0 con 0x7fb0dc1544b0 2024-03-23T15:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.521+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0dc075740 con 0x7fb0dc1544b0 2024-03-23T15:58:05.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.558+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb0d403b040 con 0x7fb0dc1544b0 2024-03-23T15:58:05.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.559+0000 7fb0c37fe640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0c806b860 0x7fb0c806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:05.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.559+0000 7fb0dad9d640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0c806b860 0x7fb0c806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:05.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.560+0000 7fb0dad9d640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0c806b860 0x7fb0c806dd20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009aa0 tx=0x7fb0cc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:05.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.812+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb0d407b1f0 con 0x7fb0dc1544b0 2024-03-23T15:58:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:05.832+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb0d4077490 con 0x7fb0dc1544b0 2024-03-23T15:58:06.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.065+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0dc0cde60 con 0x7fb0dc1544b0 2024-03-23T15:58:06.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.088+0000 7fb0c37fe640 1 -- 172.21.15.129:0/1665823836 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb0d404e090 con 0x7fb0dc1544b0 2024-03-23T15:58:06.090 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:06.091 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:06.091 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0c806b860 msgr2=0x7fb0c806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0c806b860 0x7fb0c806dd20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009aa0 tx=0x7fb0cc009190 comp rx=0 tx=0).stop 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 msgr2=0x7fb0dc0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 0x7fb0dc0cf660 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009eb0 tx=0x7fb0d4004a40 comp rx=0 tx=0).stop 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 shutdown_connections 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb0c806b860 0x7fb0c806dd20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb0dc1544b0 0x7fb0dc0cf660 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb0dc0caa00 0x7fb0dc0cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 --2- 172.21.15.129:0/1665823836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0dc0c6000 0x7fb0dc1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 >> 172.21.15.129:0/1665823836 conn(0x7fb0dc061b50 msgr2=0x7fb0dc158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:06.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 shutdown_connections 2024-03-23T15:58:06.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:06.090+0000 7fb0e107c640 1 -- 172.21.15.129:0/1665823836 wait complete. 2024-03-23T15:58:06.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:06.708 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:07.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:07.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.808+0000 7fb7f4d25640 1 Processor -- start 2024-03-23T15:58:07.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.808+0000 7fb7f4d25640 1 -- start start 2024-03-23T15:58:07.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.809+0000 7fb7f4d25640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f015dba0 0x7fb7f006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:07.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.809+0000 7fb7f4d25640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f0070640 con 0x7fb7f015df80 2024-03-23T15:58:07.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.809+0000 7fb7f4d25640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f00c4c00 con 0x7fb7f00cca30 2024-03-23T15:58:07.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.809+0000 7fb7f4d25640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f006c240 con 0x7fb7f015dba0 2024-03-23T15:58:07.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.809+0000 7fb7edd9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb7f00cca30 0x7fb7f006ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:59416/0 (socket says 172.21.15.129:59416) 2024-03-23T15:58:07.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.809+0000 7fb7edd9b640 1 -- 172.21.15.129:0/2690646057 learned_addr learned my addr 172.21.15.129:0/2690646057 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:07.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.810+0000 7fb7ee59c640 1 --2- 172.21.15.129:0/2690646057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f015dba0 0x7fb7f006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:07.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.812+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3903513825 0 0) 0x7fb7f0070640 con 0x7fb7f015df80 2024-03-23T15:58:07.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.812+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7d8003520 con 0x7fb7f015df80 2024-03-23T15:58:07.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.813+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141714043 0 0) 0x7fb7e4001160 con 0x7fb7f00cca30 2024-03-23T15:58:07.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.813+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7f0070640 con 0x7fb7f00cca30 2024-03-23T15:58:07.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.813+0000 7fb7ee59c640 1 -- 172.21.15.129:0/2690646057 >> v1:172.21.15.157:6789/0 conn(0x7fb7f00cca30 legacy=0x7fb7f006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:07.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.813+0000 7fb7ee59c640 1 -- 172.21.15.129:0/2690646057 >> v1:172.21.15.129:6789/0 conn(0x7fb7f015df80 legacy=0x7fb7f01540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:07.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.813+0000 7fb7ee59c640 1 -- 172.21.15.129:0/2690646057 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f00cce70 con 0x7fb7f015dba0 2024-03-23T15:58:07.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.815+0000 7fb7ee59c640 1 --2- 172.21.15.129:0/2690646057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f015dba0 0x7fb7f006cf70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00097d0 tx=0x7fb7e002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3446b4294ad71eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:07.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.820+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0030880 con 0x7fb7f015dba0 2024-03-23T15:58:07.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.820+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e00309e0 con 0x7fb7f015dba0 2024-03-23T15:58:07.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.820+0000 7fb7ed59a640 1 -- 172.21.15.129:0/2690646057 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0030d70 con 0x7fb7f015dba0 2024-03-23T15:58:07.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2690646057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f015dba0 msgr2=0x7fb7f006cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:07.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2690646057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f015dba0 0x7fb7f006cf70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00097d0 tx=0x7fb7e002b9c0 comp rx=0 tx=0).stop 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2690646057 shutdown_connections 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2690646057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f015dba0 0x7fb7f006cf70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2690646057 >> 172.21.15.129:0/2690646057 conn(0x7fb7f0071300 msgr2=0x7fb7f00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2690646057 shutdown_connections 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.821+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2690646057 wait complete. 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.822+0000 7fb7f4d25640 1 Processor -- start 2024-03-23T15:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.822+0000 7fb7f4d25640 1 -- start start 2024-03-23T15:58:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.823+0000 7fb7f4d25640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f00cca30 0x7fb7f0149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.823+0000 7fb7f4d25640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f015dba0 0x7fb7f0149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.823+0000 7fb7f4d25640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f015df80 0x7fb7f01f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.823+0000 7fb7f4d25640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f01f0000 con 0x7fb7f015dba0 2024-03-23T15:58:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.823+0000 7fb7f4d25640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f01f3440 con 0x7fb7f015df80 2024-03-23T15:58:07.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.823+0000 7fb7f4d25640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f00c8b10 con 0x7fb7f00cca30 2024-03-23T15:58:07.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.824+0000 7fb7ee59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f00cca30 0x7fb7f0149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:07.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.824+0000 7fb7edd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f015dba0 0x7fb7f0149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:07.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.824+0000 7fb7eed9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f015df80 0x7fb7f01f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:07.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.824+0000 7fb7ee59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f00cca30 0x7fb7f0149110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:35006/0 (socket says 172.21.15.129:35006) 2024-03-23T15:58:07.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.824+0000 7fb7ee59c640 1 -- 172.21.15.129:0/2240972815 learned_addr learned my addr 172.21.15.129:0/2240972815 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:07.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.827+0000 7fb7edd9b640 1 -- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f00cca30 msgr2=0x7fb7f0149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:07.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.827+0000 7fb7edd9b640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f00cca30 0x7fb7f0149110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:07.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.827+0000 7fb7edd9b640 1 -- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f015df80 msgr2=0x7fb7f01f0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:07.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.827+0000 7fb7edd9b640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f015df80 0x7fb7f01f0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:07.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.827+0000 7fb7edd9b640 1 -- 172.21.15.129:0/2240972815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e0009400 con 0x7fb7f015dba0 2024-03-23T15:58:07.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.830+0000 7fb7edd9b640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f015dba0 0x7fb7f0149a50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400c700 tx=0x7fb7e400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.834+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4005b00 con 0x7fb7f015dba0 2024-03-23T15:58:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.834+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e4005c60 con 0x7fb7f015dba0 2024-03-23T15:58:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.834+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f01f3700 con 0x7fb7f015dba0 2024-03-23T15:58:07.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.834+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40072b0 con 0x7fb7f015dba0 2024-03-23T15:58:07.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.834+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f01f3cc0 con 0x7fb7f015dba0 2024-03-23T15:58:07.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.835+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f01f0190 con 0x7fb7f015dba0 2024-03-23T15:58:07.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.854+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb7e4007410 con 0x7fb7f015dba0 2024-03-23T15:58:07.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.854+0000 7fb7deffd640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7d006b860 0x7fb7d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:07.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.855+0000 7fb7ee59c640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7d006b860 0x7fb7d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:07.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:07.855+0000 7fb7ee59c640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7d006b860 0x7fb7d006dd20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb7e002ba00 tx=0x7fb7e0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:08.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.103+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb7e4007d10 con 0x7fb7f015dba0 2024-03-23T15:58:08.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.103+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb7e4054430 con 0x7fb7f015dba0 2024-03-23T15:58:08.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.321+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7f00c4c00 con 0x7fb7f015dba0 2024-03-23T15:58:08.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.341+0000 7fb7deffd640 1 -- 172.21.15.129:0/2240972815 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb7f00c4c00 con 0x7fb7f015dba0 2024-03-23T15:58:08.344 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:08.344 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:08.344 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7d006b860 msgr2=0x7fb7d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7d006b860 0x7fb7d006dd20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb7e002ba00 tx=0x7fb7e0008030 comp rx=0 tx=0).stop 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f015dba0 msgr2=0x7fb7f0149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f015dba0 0x7fb7f0149a50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400c700 tx=0x7fb7e400cbc0 comp rx=0 tx=0).stop 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 shutdown_connections 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7d006b860 0x7fb7d006dd20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f015df80 0x7fb7f01f0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f015dba0 0x7fb7f0149a50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 --2- 172.21.15.129:0/2240972815 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7f00cca30 0x7fb7f0149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:08.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 >> 172.21.15.129:0/2240972815 conn(0x7fb7f0071300 msgr2=0x7fb7f00c9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:08.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 shutdown_connections 2024-03-23T15:58:08.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:08.343+0000 7fb7f4d25640 1 -- 172.21.15.129:0/2240972815 wait complete. 2024-03-23T15:58:08.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:58:08.702 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:58:08.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:08.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:09.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:10.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.063+0000 7f8534639640 1 Processor -- start 2024-03-23T15:58:10.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.063+0000 7f8534639640 1 -- start start 2024-03-23T15:58:10.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.064+0000 7f8534639640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c06c030 0x7f852c06c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:10.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.064+0000 7f8534639640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f852c0cc080 0x7f852c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:10.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.064+0000 7f8534639640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f852c06cae0 con 0x7f852c0cbc60 2024-03-23T15:58:10.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.064+0000 7f8534639640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f852c0cabc0 con 0x7f852c0cc080 2024-03-23T15:58:10.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.064+0000 7f8534639640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f852c0caea0 con 0x7f852c06c030 2024-03-23T15:58:10.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.065+0000 7f85323d5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c06c030 0x7f852c06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:10.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.065+0000 7f8532bd6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f852c0cc080 0x7f852c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:10.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.065+0000 7f8531bd4640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f852c0cbc60 0x7f852c15d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:59426/0 (socket says 172.21.15.129:59426) 2024-03-23T15:58:10.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.065+0000 7f8531bd4640 1 -- 172.21.15.129:0/3940465742 learned_addr learned my addr 172.21.15.129:0/3940465742 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:10.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.068+0000 7f85323d5640 1 -- 172.21.15.129:0/3940465742 >> v2:172.21.15.157:3300/0 conn(0x7f852c0cc080 msgr2=0x7f852c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:10.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.068+0000 7f85323d5640 1 --2- 172.21.15.129:0/3940465742 >> v2:172.21.15.157:3300/0 conn(0x7f852c0cc080 0x7f852c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.068+0000 7f85323d5640 1 -- 172.21.15.129:0/3940465742 >> v1:172.21.15.157:6789/0 conn(0x7f852c0cbc60 legacy=0x7f852c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:10.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.068+0000 7f85323d5640 1 -- 172.21.15.129:0/3940465742 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852c0cb180 con 0x7f852c06c030 2024-03-23T15:58:10.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.070+0000 7f85323d5640 1 --2- 172.21.15.129:0/3940465742 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c06c030 0x7f852c06c410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f85280097a0 tx=0x7f852802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98b3303255782e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.074+0000 7f85313d3640 1 -- 172.21.15.129:0/3940465742 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852802e070 con 0x7f852c06c030 2024-03-23T15:58:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.074+0000 7f85313d3640 1 -- 172.21.15.129:0/3940465742 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85280315a0 con 0x7f852c06c030 2024-03-23T15:58:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.074+0000 7f85313d3640 1 -- 172.21.15.129:0/3940465742 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528031930 con 0x7f852c06c030 2024-03-23T15:58:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 -- 172.21.15.129:0/3940465742 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c06c030 msgr2=0x7f852c06c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 --2- 172.21.15.129:0/3940465742 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c06c030 0x7f852c06c410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f85280097a0 tx=0x7f852802b840 comp rx=0 tx=0).stop 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 -- 172.21.15.129:0/3940465742 shutdown_connections 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 --2- 172.21.15.129:0/3940465742 >> v2:172.21.15.157:3300/0 conn(0x7f852c0cc080 0x7f852c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 --2- 172.21.15.129:0/3940465742 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c06c030 0x7f852c06c410 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 -- 172.21.15.129:0/3940465742 >> 172.21.15.129:0/3940465742 conn(0x7f852c0cfb20 msgr2=0x7f852c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 -- 172.21.15.129:0/3940465742 shutdown_connections 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.075+0000 7f8534639640 1 -- 172.21.15.129:0/3940465742 wait complete. 2024-03-23T15:58:10.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.076+0000 7f8534639640 1 Processor -- start 2024-03-23T15:58:10.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.076+0000 7f8534639640 1 -- start start 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.077+0000 7f8534639640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f852c06c030 0x7f852c1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.077+0000 7f8534639640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c0cbc60 0x7f852c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.077+0000 7f8534639640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f852c0cc080 0x7f852c0b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.077+0000 7f8534639640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f852c0b5500 con 0x7f852c0cc080 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.077+0000 7f8534639640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f852c1f4380 con 0x7f852c06c030 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.077+0000 7f8534639640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f852c0b3fa0 con 0x7f852c0cbc60 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.078+0000 7f85323d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f852c06c030 0x7f852c1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:10.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.078+0000 7f8531bd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c0cbc60 0x7f852c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:10.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.078+0000 7f85323d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f852c06c030 0x7f852c1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:51776/0 (socket says 172.21.15.129:51776) 2024-03-23T15:58:10.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.078+0000 7f85323d5640 1 -- 172.21.15.129:0/3205582179 learned_addr learned my addr 172.21.15.129:0/3205582179 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.082+0000 7f8531bd4640 1 -- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f852c06c030 msgr2=0x7f852c1f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.082+0000 7f8531bd4640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f852c06c030 0x7f852c1f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.082+0000 7f8531bd4640 1 -- 172.21.15.129:0/3205582179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f852c0cc080 msgr2=0x7f852c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.082+0000 7f8531bd4640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f852c0cc080 0x7f852c0b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.082+0000 7f8531bd4640 1 -- 172.21.15.129:0/3205582179 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8528009400 con 0x7f852c0cbc60 2024-03-23T15:58:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.085+0000 7f8531bd4640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c0cbc60 0x7f852c0b33a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f851c00c370 tx=0x7f851c00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:10.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.089+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c005ce0 con 0x7f852c0cbc60 2024-03-23T15:58:10.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.089+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f851c005e40 con 0x7f852c0cbc60 2024-03-23T15:58:10.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.089+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852c1f03e0 con 0x7f852c0cbc60 2024-03-23T15:58:10.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.089+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c007310 con 0x7f852c0cbc60 2024-03-23T15:58:10.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.089+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c1f09a0 con 0x7f852c0cbc60 2024-03-23T15:58:10.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.091+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f852c075740 con 0x7f852c0cbc60 2024-03-23T15:58:10.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.122+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f851c007470 con 0x7f852c0cbc60 2024-03-23T15:58:10.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.123+0000 7f8522ffd640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f851006b860 0x7f851006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:10.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.123+0000 7f85323d5640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f851006b860 0x7f851006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:10.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.124+0000 7f85323d5640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f851006b860 0x7f851006dd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f852802b880 tx=0x7f852803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.403+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f851c007720 con 0x7f852c0cbc60 2024-03-23T15:58:10.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.421+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f851c054840 con 0x7f852c0cbc60 2024-03-23T15:58:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.645+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f852c0d0a30 con 0x7f852c0cbc60 2024-03-23T15:58:10.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.755+0000 7f8522ffd640 1 -- 172.21.15.129:0/3205582179 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f851c017020 con 0x7f852c0cbc60 2024-03-23T15:58:10.757 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:10.757 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:10.758 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:10.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f851006b860 msgr2=0x7f851006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:10.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f851006b860 0x7f851006dd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f852802b880 tx=0x7f852803d040 comp rx=0 tx=0).stop 2024-03-23T15:58:10.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c0cbc60 msgr2=0x7f852c0b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c0cbc60 0x7f852c0b33a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f851c00c370 tx=0x7f851c00c830 comp rx=0 tx=0).stop 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 shutdown_connections 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f852c0cc080 0x7f852c0b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f852c0cbc60 0x7f852c0b33a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f851006b860 0x7f851006dd20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 --2- 172.21.15.129:0/3205582179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f852c06c030 0x7f852c1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 >> 172.21.15.129:0/3205582179 conn(0x7f852c0cfb20 msgr2=0x7f852c0af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 shutdown_connections 2024-03-23T15:58:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:10.757+0000 7f8534639640 1 -- 172.21.15.129:0/3205582179 wait complete. 2024-03-23T15:58:11.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:11.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:12.379 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:12.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.479+0000 7f59a15e2640 1 Processor -- start 2024-03-23T15:58:12.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.479+0000 7f59a15e2640 1 -- start start 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.480+0000 7f59a15e2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f599c15a2e0 0x7f599c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.480+0000 7f59a15e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.480+0000 7f59a15e2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f599c0cbcb0 con 0x7f599c1544b0 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.480+0000 7f59a15e2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f599c0c5be0 con 0x7f599c15a2e0 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.480+0000 7f59a15e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f599c15b280 con 0x7f599c06c0b0 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.481+0000 7f599a7fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f599c15a2e0 0x7f599c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.481+0000 7f599b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:12.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.481+0000 7f599a7fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f599c15a2e0 0x7f599c06bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49196/0 (socket says 172.21.15.129:49196) 2024-03-23T15:58:12.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.481+0000 7f599a7fc640 1 -- 172.21.15.129:0/133285972 learned_addr learned my addr 172.21.15.129:0/133285972 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:12.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.484+0000 7f599a7fc640 1 -- 172.21.15.129:0/133285972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 msgr2=0x7f599c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:12.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.484+0000 7f599a7fc640 1 --2- 172.21.15.129:0/133285972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c0c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:12.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.484+0000 7f599a7fc640 1 -- 172.21.15.129:0/133285972 >> v1:172.21.15.129:6789/0 conn(0x7f599c1544b0 legacy=0x7f599c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:12.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.484+0000 7f599a7fc640 1 -- 172.21.15.129:0/133285972 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c15a9e0 con 0x7f599c15a2e0 2024-03-23T15:58:12.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.487+0000 7f599a7fc640 1 --2- 172.21.15.129:0/133285972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c06bb70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5990009870 tx=0x7f599002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9a54ee2815160f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:12.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.496+0000 7f5999ffb640 1 -- 172.21.15.129:0/133285972 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599002f840 con 0x7f599c15a2e0 2024-03-23T15:58:12.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f5999ffb640 1 -- 172.21.15.129:0/133285972 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f599002f9a0 con 0x7f599c15a2e0 2024-03-23T15:58:12.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f5999ffb640 1 -- 172.21.15.129:0/133285972 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599002fd30 con 0x7f599c15a2e0 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f59a15e2640 1 -- 172.21.15.129:0/133285972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 msgr2=0x7f599c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f59a15e2640 1 --2- 172.21.15.129:0/133285972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c06bb70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5990009870 tx=0x7f599002ba20 comp rx=0 tx=0).stop 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f59a15e2640 1 -- 172.21.15.129:0/133285972 shutdown_connections 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f59a15e2640 1 --2- 172.21.15.129:0/133285972 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f59a15e2640 1 --2- 172.21.15.129:0/133285972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c06bb70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.499+0000 7f59a15e2640 1 -- 172.21.15.129:0/133285972 >> 172.21.15.129:0/133285972 conn(0x7f599c061b50 msgr2=0x7f599c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:12.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.500+0000 7f59a15e2640 1 -- 172.21.15.129:0/133285972 shutdown_connections 2024-03-23T15:58:12.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.500+0000 7f59a15e2640 1 -- 172.21.15.129:0/133285972 wait complete. 2024-03-23T15:58:12.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.501+0000 7f59a15e2640 1 Processor -- start 2024-03-23T15:58:12.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.501+0000 7f59a15e2640 1 -- start start 2024-03-23T15:58:12.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f59a15e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c1f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:12.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f59a15e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f599c1544b0 0x7f599c1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:12.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f59a15e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c14b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:12.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f59a15e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1f1540 con 0x7f599c15a2e0 2024-03-23T15:58:12.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f59a15e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f599c14c3a0 con 0x7f599c1544b0 2024-03-23T15:58:12.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f59a15e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f599c0d0060 con 0x7f599c06c0b0 2024-03-23T15:58:12.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.502+0000 7f599affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c1f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:12.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.503+0000 7f599a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f599c1544b0 0x7f599c1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:12.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.503+0000 7f599affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c1f1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50306/0 (socket says 172.21.15.129:50306) 2024-03-23T15:58:12.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.503+0000 7f599affd640 1 -- 172.21.15.129:0/656974377 learned_addr learned my addr 172.21.15.129:0/656974377 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:12.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.504+0000 7f599b7fe640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c14b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.507+0000 7f599a7fc640 1 -- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 msgr2=0x7f599c1f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.507+0000 7f599a7fc640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c1f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.507+0000 7f599a7fc640 1 -- 172.21.15.129:0/656974377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 msgr2=0x7f599c14b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.507+0000 7f599a7fc640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c14b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:12.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.507+0000 7f599a7fc640 1 -- 172.21.15.129:0/656974377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59900094d0 con 0x7f599c1544b0 2024-03-23T15:58:12.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.509+0000 7f599a7fc640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f599c1544b0 0x7f599c1f43b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5990033000 tx=0x7f5990004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.514+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990037bc0 con 0x7f599c1544b0 2024-03-23T15:58:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.514+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5990037d40 con 0x7f599c1544b0 2024-03-23T15:58:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.514+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599c0d0340 con 0x7f599c1544b0 2024-03-23T15:58:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.514+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990041640 con 0x7f599c1544b0 2024-03-23T15:58:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.514+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f599c0d0900 con 0x7f599c1544b0 2024-03-23T15:58:12.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.515+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f599c0ca770 con 0x7f599c1544b0 2024-03-23T15:58:12.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.545+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f59900417a0 con 0x7f599c1544b0 2024-03-23T15:58:12.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.546+0000 7f59837fe640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f598806b860 0x7f598806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:12.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.546+0000 7f599affd640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f598806b860 0x7f598806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:12.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.546+0000 7f599affd640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f598806b860 0x7f598806dd20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f598c001840 tx=0x7f598c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.549+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f599003c070 con 0x7f599c1544b0 2024-03-23T15:58:12.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:12.813+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f59900782e0 con 0x7f599c1544b0 2024-03-23T15:58:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.035+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f599c0cb2d0 con 0x7f599c1544b0 2024-03-23T15:58:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.150+0000 7f59837fe640 1 -- 172.21.15.129:0/656974377 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f599007b9a0 con 0x7f599c1544b0 2024-03-23T15:58:13.153 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:13.153 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f598806b860 msgr2=0x7f598806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f598806b860 0x7f598806dd20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f598c001840 tx=0x7f598c009040 comp rx=0 tx=0).stop 2024-03-23T15:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f599c1544b0 msgr2=0x7f599c1f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f599c1544b0 0x7f599c1f43b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5990033000 tx=0x7f5990004730 comp rx=0 tx=0).stop 2024-03-23T15:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 shutdown_connections 2024-03-23T15:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f599c15a2e0 0x7f599c14b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f598806b860 0x7f598806dd20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f599c1544b0 0x7f599c1f43b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 --2- 172.21.15.129:0/656974377 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f599c06c0b0 0x7f599c1f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 >> 172.21.15.129:0/656974377 conn(0x7f599c061b50 msgr2=0x7f599c158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 shutdown_connections 2024-03-23T15:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:13.153+0000 7f59a15e2640 1 -- 172.21.15.129:0/656974377 wait complete. 2024-03-23T15:58:13.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:13.770 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:14.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:14.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.871+0000 7fee3a6c3640 1 Processor -- start 2024-03-23T15:58:14.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.872+0000 7fee3a6c3640 1 -- start start 2024-03-23T15:58:14.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee3a6c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee3406c030 0x7fee3406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:14.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee3a6c3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee3406cae0 con 0x7fee340cc080 2024-03-23T15:58:14.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee3a6c3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee340cabc0 con 0x7fee340cbc60 2024-03-23T15:58:14.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee3a6c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fee3415e720 con 0x7fee3406c030 2024-03-23T15:58:14.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee38c60640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fee340cc080 0x7fee3415e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55478/0 (socket says 172.21.15.129:55478) 2024-03-23T15:58:14.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee38c60640 1 -- 172.21.15.129:0/3557322944 learned_addr learned my addr 172.21.15.129:0/3557322944 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:14.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.873+0000 7fee33fff640 1 --2- 172.21.15.129:0/3557322944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee3406c030 0x7fee3406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:14.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.876+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2285988045 0 0) 0x7fee3406cae0 con 0x7fee340cc080 2024-03-23T15:58:14.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.877+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee1c003520 con 0x7fee340cc080 2024-03-23T15:58:14.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.877+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847119854 0 0) 0x7fee340cabc0 con 0x7fee340cbc60 2024-03-23T15:58:14.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.877+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee3406cae0 con 0x7fee340cbc60 2024-03-23T15:58:14.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.877+0000 7fee33fff640 1 -- 172.21.15.129:0/3557322944 >> v1:172.21.15.157:6789/0 conn(0x7fee340cbc60 legacy=0x7fee3415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:14.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.877+0000 7fee33fff640 1 -- 172.21.15.129:0/3557322944 >> v1:172.21.15.129:6789/0 conn(0x7fee340cc080 legacy=0x7fee3415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:14.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.877+0000 7fee33fff640 1 -- 172.21.15.129:0/3557322944 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee340caea0 con 0x7fee3406c030 2024-03-23T15:58:14.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.880+0000 7fee33fff640 1 --2- 172.21.15.129:0/3557322944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee3406c030 0x7fee3406c410 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee240097d0 tx=0x7fee2402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47e96d503c6ad97c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:14.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24031800 con 0x7fee3406c030 2024-03-23T15:58:14.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee24031960 con 0x7fee3406c030 2024-03-23T15:58:14.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee32ffd640 1 -- 172.21.15.129:0/3557322944 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24031cf0 con 0x7fee3406c030 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 -- 172.21.15.129:0/3557322944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee3406c030 msgr2=0x7fee3406c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/3557322944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee3406c030 0x7fee3406c410 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee240097d0 tx=0x7fee2402b8a0 comp rx=0 tx=0).stop 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 -- 172.21.15.129:0/3557322944 shutdown_connections 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/3557322944 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee3406c030 0x7fee3406c410 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 -- 172.21.15.129:0/3557322944 >> 172.21.15.129:0/3557322944 conn(0x7fee340cfb20 msgr2=0x7fee340cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 -- 172.21.15.129:0/3557322944 shutdown_connections 2024-03-23T15:58:14.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.885+0000 7fee3a6c3640 1 -- 172.21.15.129:0/3557322944 wait complete. 2024-03-23T15:58:14.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.887+0000 7fee3a6c3640 1 Processor -- start 2024-03-23T15:58:14.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.887+0000 7fee3a6c3640 1 -- start start 2024-03-23T15:58:14.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.887+0000 7fee3a6c3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee3406c030 0x7fee34157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:14.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.887+0000 7fee3a6c3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee340cbc60 0x7fee341f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:14.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.887+0000 7fee3a6c3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee340cc080 0x7fee341f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:14.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee3a6c3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee341f0d20 con 0x7fee340cc080 2024-03-23T15:58:14.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee3a6c3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fee3414f7d0 con 0x7fee3406c030 2024-03-23T15:58:14.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee3a6c3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fee341f0780 con 0x7fee340cbc60 2024-03-23T15:58:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee38c60640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee340cc080 0x7fee341f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee337fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee340cbc60 0x7fee341f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee33fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee3406c030 0x7fee34157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee337fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee340cbc60 0x7fee341f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50330/0 (socket says 172.21.15.129:50330) 2024-03-23T15:58:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.888+0000 7fee337fe640 1 -- 172.21.15.129:0/1496475970 learned_addr learned my addr 172.21.15.129:0/1496475970 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:14.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.892+0000 7fee38c60640 1 -- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee340cbc60 msgr2=0x7fee341f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:14.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.892+0000 7fee38c60640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee340cbc60 0x7fee341f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:14.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.892+0000 7fee38c60640 1 -- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee3406c030 msgr2=0x7fee34157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:14.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.892+0000 7fee38c60640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee3406c030 0x7fee34157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:14.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.892+0000 7fee38c60640 1 -- 172.21.15.129:0/1496475970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee24009400 con 0x7fee340cc080 2024-03-23T15:58:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.894+0000 7fee38c60640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee340cc080 0x7fee341f37b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00ca70 tx=0x7fee2c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:14.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.898+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00abd0 con 0x7fee340cc080 2024-03-23T15:58:14.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.899+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2c00ad30 con 0x7fee340cc080 2024-03-23T15:58:14.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.899+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c012640 con 0x7fee340cc080 2024-03-23T15:58:14.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.899+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee341f0a40 con 0x7fee340cc080 2024-03-23T15:58:14.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.899+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3414f110 con 0x7fee340cc080 2024-03-23T15:58:14.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.900+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee340686f0 con 0x7fee340cc080 2024-03-23T15:58:14.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.923+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee2c007070 con 0x7fee340cc080 2024-03-23T15:58:14.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.923+0000 7fee30ff9640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fee2006b810 0x7fee2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:14.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.923+0000 7fee33fff640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fee2006b810 0x7fee2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:14.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.924+0000 7fee33fff640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fee2006b810 0x7fee2006dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fee24036000 tx=0x7fee2403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:14.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:14.929+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fee2c058d30 con 0x7fee340cc080 2024-03-23T15:58:15.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.191+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee2c054fd0 con 0x7fee340cc080 2024-03-23T15:58:15.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.484+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee3414f910 con 0x7fee340cc080 2024-03-23T15:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.497+0000 7fee30ff9640 1 -- 172.21.15.129:0/1496475970 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fee3414f910 con 0x7fee340cc080 2024-03-23T15:58:15.499 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:15.499 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:15.500 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fee2006b810 msgr2=0x7fee2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fee2006b810 0x7fee2006dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fee24036000 tx=0x7fee2403c040 comp rx=0 tx=0).stop 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee340cc080 msgr2=0x7fee341f37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee340cc080 0x7fee341f37b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00ca70 tx=0x7fee2c00cf30 comp rx=0 tx=0).stop 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 shutdown_connections 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee340cc080 0x7fee341f37b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee340cbc60 0x7fee341f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fee2006b810 0x7fee2006dcd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 --2- 172.21.15.129:0/1496475970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee3406c030 0x7fee34157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 >> 172.21.15.129:0/1496475970 conn(0x7fee340cfb20 msgr2=0x7fee340c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 shutdown_connections 2024-03-23T15:58:15.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:15.499+0000 7fee3a6c3640 1 -- 172.21.15.129:0/1496475970 wait complete. 2024-03-23T15:58:16.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:16.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:17.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:17.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 Processor -- start 2024-03-23T15:58:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 -- start start 2024-03-23T15:58:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f65b01544b0 0x7f65b015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65b00c6000 0x7f65b00c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b00cbcb0 con 0x7f65b00caa00 2024-03-23T15:58:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f65b00c5be0 con 0x7f65b00c6000 2024-03-23T15:58:17.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.221+0000 7f65b6d9a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f65b00c69b0 con 0x7f65b01544b0 2024-03-23T15:58:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.222+0000 7f65b4b36640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f65b01544b0 0x7f65b015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.222+0000 7f65b5337640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f65b00c6000 0x7f65b00c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.223+0000 7f65b4b36640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f65b01544b0 0x7f65b015a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43354/0 (socket says 172.21.15.129:43354) 2024-03-23T15:58:17.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.223+0000 7f65b4b36640 1 -- 172.21.15.129:0/390275160 learned_addr learned my addr 172.21.15.129:0/390275160 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.226+0000 7f65b4b36640 1 -- 172.21.15.129:0/390275160 >> v2:172.21.15.129:3300/0 conn(0x7f65b00c6000 msgr2=0x7f65b00c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.226+0000 7f65b4b36640 1 --2- 172.21.15.129:0/390275160 >> v2:172.21.15.129:3300/0 conn(0x7f65b00c6000 0x7f65b00c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.226+0000 7f65b4b36640 1 -- 172.21.15.129:0/390275160 >> v1:172.21.15.129:6789/0 conn(0x7f65b00caa00 legacy=0x7f65b006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.226+0000 7f65b4b36640 1 -- 172.21.15.129:0/390275160 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b015b280 con 0x7f65b01544b0 2024-03-23T15:58:17.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.229+0000 7f65b4b36640 1 --2- 172.21.15.129:0/390275160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b015a2e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f65b006cd70 tx=0x7f65a402ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96465c7871aaca05 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:17.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65af7fe640 1 -- 172.21.15.129:0/390275160 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a402f850 con 0x7f65b01544b0 2024-03-23T15:58:17.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65af7fe640 1 -- 172.21.15.129:0/390275160 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a402f9b0 con 0x7f65b01544b0 2024-03-23T15:58:17.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65af7fe640 1 -- 172.21.15.129:0/390275160 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a402fd40 con 0x7f65b01544b0 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 -- 172.21.15.129:0/390275160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 msgr2=0x7f65b015a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/390275160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b015a2e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f65b006cd70 tx=0x7f65a402ba30 comp rx=0 tx=0).stop 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 -- 172.21.15.129:0/390275160 shutdown_connections 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/390275160 >> v2:172.21.15.129:3300/0 conn(0x7f65b00c6000 0x7f65b00c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/390275160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b015a2e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 -- 172.21.15.129:0/390275160 >> 172.21.15.129:0/390275160 conn(0x7f65b0061b50 msgr2=0x7f65b0061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 -- 172.21.15.129:0/390275160 shutdown_connections 2024-03-23T15:58:17.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.235+0000 7f65b6d9a640 1 -- 172.21.15.129:0/390275160 wait complete. 2024-03-23T15:58:17.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 Processor -- start 2024-03-23T15:58:17.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 -- start start 2024-03-23T15:58:17.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65b00c6000 0x7f65b01f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:17.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b00caa00 0x7f65b00c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:17.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b01594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:17.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f65b01f0c50 con 0x7f65b00caa00 2024-03-23T15:58:17.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65b00c6f00 con 0x7f65b01544b0 2024-03-23T15:58:17.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.237+0000 7f65b6d9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f65b00c74c0 con 0x7f65b00c6000 2024-03-23T15:58:17.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.238+0000 7f65b5337640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b01594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:17.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.238+0000 7f65affff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b00caa00 0x7f65b00c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:17.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.238+0000 7f65b4b36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65b00c6000 0x7f65b01f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:17.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.238+0000 7f65b5337640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b01594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43362/0 (socket says 172.21.15.129:43362) 2024-03-23T15:58:17.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.238+0000 7f65b5337640 1 -- 172.21.15.129:0/3212893139 learned_addr learned my addr 172.21.15.129:0/3212893139 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:17.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.242+0000 7f65b4b36640 1 -- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 msgr2=0x7f65b01594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:17.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.242+0000 7f65b4b36640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b01594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.242+0000 7f65b4b36640 1 -- 172.21.15.129:0/3212893139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b00caa00 msgr2=0x7f65b00c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:17.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.242+0000 7f65b4b36640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b00caa00 0x7f65b00c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.242+0000 7f65b4b36640 1 -- 172.21.15.129:0/3212893139 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a40094d0 con 0x7f65b00c6000 2024-03-23T15:58:17.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.245+0000 7f65b4b36640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65b00c6000 0x7f65b01f0710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f65a4033000 tx=0x7f65a4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.247+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4037c40 con 0x7f65b00c6000 2024-03-23T15:58:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.247+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b0154c70 con 0x7f65b00c6000 2024-03-23T15:58:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.247+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b00c7e90 con 0x7f65b00c6000 2024-03-23T15:58:17.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.248+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b0075740 con 0x7f65b00c6000 2024-03-23T15:58:17.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.249+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a4037da0 con 0x7f65b00c6000 2024-03-23T15:58:17.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.249+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a404b660 con 0x7f65b00c6000 2024-03-23T15:58:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.504+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f65a404b7c0 con 0x7f65b00c6000 2024-03-23T15:58:17.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.504+0000 7f65ad7fa640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f659406b810 0x7f659406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.504+0000 7f65affff640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f659406b810 0x7f659406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:17.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.505+0000 7f65affff640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f659406b810 0x7f659406dcd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f65b0061fa0 tx=0x7f659c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:17.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.508+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f65a403c070 con 0x7f65b00c6000 2024-03-23T15:58:17.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.520+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f65a4078580 con 0x7f65b00c6000 2024-03-23T15:58:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.766+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65b00cb540 con 0x7f65b00c6000 2024-03-23T15:58:17.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.791+0000 7f65ad7fa640 1 -- 172.21.15.129:0/3212893139 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f65a407bc40 con 0x7f65b00c6000 2024-03-23T15:58:17.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:17.793 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:17.793 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f659406b810 msgr2=0x7f659406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f659406b810 0x7f659406dcd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f65b0061fa0 tx=0x7f659c009040 comp rx=0 tx=0).stop 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65b00c6000 msgr2=0x7f65b01f0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65b00c6000 0x7f65b01f0710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f65a4033000 tx=0x7f65a4004730 comp rx=0 tx=0).stop 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 shutdown_connections 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f659406b810 0x7f659406dcd0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65b01544b0 0x7f65b01594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.793+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b00caa00 0x7f65b00c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.794+0000 7f65b6d9a640 1 --2- 172.21.15.129:0/3212893139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65b00c6000 0x7f65b01f0710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:17.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.794+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 >> 172.21.15.129:0/3212893139 conn(0x7f65b0061b50 msgr2=0x7f65b00cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:17.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.794+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 shutdown_connections 2024-03-23T15:58:17.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:17.794+0000 7f65b6d9a640 1 -- 172.21.15.129:0/3212893139 wait complete. 2024-03-23T15:58:18.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:18.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:19.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:19.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.515+0000 7f173462c640 1 Processor -- start 2024-03-23T15:58:19.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.515+0000 7f173462c640 1 -- start start 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.516+0000 7f173462c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f172c0cdc00 0x7f172c06fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.516+0000 7f173462c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f172c0cba60 con 0x7f172c0cd820 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.516+0000 7f173462c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f172c0cd120 con 0x7f172c0ce050 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.516+0000 7f173462c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f172c070e10 con 0x7f172c0cdc00 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.517+0000 7f1731bc7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f172c0cdc00 0x7f172c06fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.517+0000 7f1732bc9640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f172c0ce050 0x7f172c151e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:46520/0 (socket says 172.21.15.129:46520) 2024-03-23T15:58:19.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.517+0000 7f1732bc9640 1 -- 172.21.15.129:0/1272279532 learned_addr learned my addr 172.21.15.129:0/1272279532 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:19.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.521+0000 7f17313c6640 1 -- 172.21.15.129:0/1272279532 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859050514 0 0) 0x7f172c0cd120 con 0x7f172c0ce050 2024-03-23T15:58:19.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.521+0000 7f17313c6640 1 -- 172.21.15.129:0/1272279532 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1718003520 con 0x7f172c0ce050 2024-03-23T15:58:19.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.521+0000 7f1731bc7640 1 -- 172.21.15.129:0/1272279532 >> v1:172.21.15.157:6789/0 conn(0x7f172c0ce050 legacy=0x7f172c151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:19.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.521+0000 7f1731bc7640 1 -- 172.21.15.129:0/1272279532 >> v1:172.21.15.129:6789/0 conn(0x7f172c0cd820 legacy=0x7f172c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:19.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.521+0000 7f1731bc7640 1 -- 172.21.15.129:0/1272279532 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172c070b30 con 0x7f172c0cdc00 2024-03-23T15:58:19.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.525+0000 7f1731bc7640 1 --2- 172.21.15.129:0/1272279532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cdc00 0x7f172c06fb30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f171c009790 tx=0x7f171c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957e6d9be8a4e7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:19.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.529+0000 7f17313c6640 1 -- 172.21.15.129:0/1272279532 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c030610 con 0x7f172c0cdc00 2024-03-23T15:58:19.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.529+0000 7f17313c6640 1 -- 172.21.15.129:0/1272279532 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171c030770 con 0x7f172c0cdc00 2024-03-23T15:58:19.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.529+0000 7f17313c6640 1 -- 172.21.15.129:0/1272279532 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c030b00 con 0x7f172c0cdc00 2024-03-23T15:58:19.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 -- 172.21.15.129:0/1272279532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cdc00 msgr2=0x7f172c06fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 --2- 172.21.15.129:0/1272279532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cdc00 0x7f172c06fb30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f171c009790 tx=0x7f171c02b960 comp rx=0 tx=0).stop 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 -- 172.21.15.129:0/1272279532 shutdown_connections 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 --2- 172.21.15.129:0/1272279532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cdc00 0x7f172c06fb30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 -- 172.21.15.129:0/1272279532 >> 172.21.15.129:0/1272279532 conn(0x7f172c071300 msgr2=0x7f172c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 -- 172.21.15.129:0/1272279532 shutdown_connections 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.530+0000 7f173462c640 1 -- 172.21.15.129:0/1272279532 wait complete. 2024-03-23T15:58:19.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.531+0000 7f173462c640 1 Processor -- start 2024-03-23T15:58:19.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.531+0000 7f173462c640 1 -- start start 2024-03-23T15:58:19.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.532+0000 7f173462c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cd820 0x7f172c06de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:19.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.532+0000 7f173462c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f172c0cdc00 0x7f172c06cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.532+0000 7f173462c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172c0ce050 0x7f172c1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.532+0000 7f173462c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f172c06d260 con 0x7f172c0cd820 2024-03-23T15:58:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.532+0000 7f173462c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f172c0c52c0 con 0x7f172c0ce050 2024-03-23T15:58:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.532+0000 7f173462c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f172c0c5580 con 0x7f172c0cdc00 2024-03-23T15:58:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.533+0000 7f1732bc9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172c0ce050 0x7f172c1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:19.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.533+0000 7f17323c8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cd820 0x7f172c06de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:19.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.533+0000 7f1731bc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f172c0cdc00 0x7f172c06cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:19.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.533+0000 7f1732bc9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172c0ce050 0x7f172c1571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43364/0 (socket says 172.21.15.129:43364) 2024-03-23T15:58:19.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.533+0000 7f1732bc9640 1 -- 172.21.15.129:0/3027455518 learned_addr learned my addr 172.21.15.129:0/3027455518 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.537+0000 7f17323c8640 1 -- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f172c0cdc00 msgr2=0x7f172c06cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.537+0000 7f17323c8640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f172c0cdc00 0x7f172c06cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.537+0000 7f17323c8640 1 -- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172c0ce050 msgr2=0x7f172c1571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.537+0000 7f17323c8640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172c0ce050 0x7f172c1571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.537+0000 7f17323c8640 1 -- 172.21.15.129:0/3027455518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c0093a0 con 0x7f172c0cd820 2024-03-23T15:58:19.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.540+0000 7f17323c8640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cd820 0x7f172c06de90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f172800c3c0 tx=0x7f172800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:19.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.546+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17280052d0 con 0x7f172c0cd820 2024-03-23T15:58:19.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.546+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1728005450 con 0x7f172c0cd820 2024-03-23T15:58:19.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.546+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172c0c5860 con 0x7f172c0cd820 2024-03-23T15:58:19.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.546+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728007260 con 0x7f172c0cd820 2024-03-23T15:58:19.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.547+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f172c06c240 con 0x7f172c0cd820 2024-03-23T15:58:19.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.548+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f172c06cb70 con 0x7f172c0cd820 2024-03-23T15:58:19.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.567+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f17280073c0 con 0x7f172c0cd820 2024-03-23T15:58:19.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.567+0000 7f1722ffd640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f171006b890 0x7f171006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:19.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.568+0000 7f1731bc7640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f171006b890 0x7f171006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:19.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.568+0000 7f1731bc7640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f171006b890 0x7f171006dd50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f171c036000 tx=0x7f171c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.816+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1728058440 con 0x7f172c0cd820 2024-03-23T15:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:19.816+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f17280546e0 con 0x7f172c0cd820 2024-03-23T15:58:20.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.067+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f172c075740 con 0x7f172c0cd820 2024-03-23T15:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.081+0000 7f1722ffd640 1 -- 172.21.15.129:0/3027455518 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1728020020 con 0x7f172c0cd820 2024-03-23T15:58:20.084 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:20.084 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:20.084 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f171006b890 msgr2=0x7f171006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f171006b890 0x7f171006dd50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f171c036000 tx=0x7f171c008030 comp rx=0 tx=0).stop 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cd820 msgr2=0x7f172c06de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cd820 0x7f172c06de90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f172800c3c0 tx=0x7f172800c880 comp rx=0 tx=0).stop 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 shutdown_connections 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f171006b890 0x7f171006dd50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172c0ce050 0x7f172c1571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:20.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f172c0cdc00 0x7f172c06cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:20.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 --2- 172.21.15.129:0/3027455518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172c0cd820 0x7f172c06de90 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:20.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 >> 172.21.15.129:0/3027455518 conn(0x7f172c071300 msgr2=0x7f172c06fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:20.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 shutdown_connections 2024-03-23T15:58:20.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:20.083+0000 7f173462c640 1 -- 172.21.15.129:0/3027455518 wait complete. 2024-03-23T15:58:20.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:20.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:21.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:21.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.802+0000 7f3a29e9a640 1 Processor -- start 2024-03-23T15:58:21.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.802+0000 7f3a29e9a640 1 -- start start 2024-03-23T15:58:21.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.803+0000 7f3a29e9a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a240ceb00 0x7f3a2415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:21.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.803+0000 7f3a29e9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a2415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:21.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.803+0000 7f3a29e9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a2415b290 con 0x7f3a240ccd60 2024-03-23T15:58:21.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.803+0000 7f3a29e9a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3a240ce800 con 0x7f3a240ceb00 2024-03-23T15:58:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.803+0000 7f3a29e9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a24154a70 con 0x7f3a240cef40 2024-03-23T15:58:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.804+0000 7f3a23fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a2415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.804+0000 7f3a237fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3a240ccd60 0x7f3a2406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44366/0 (socket says 172.21.15.129:44366) 2024-03-23T15:58:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.804+0000 7f3a237fe640 1 -- 172.21.15.129:0/3391094333 learned_addr learned my addr 172.21.15.129:0/3391094333 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.805+0000 7f3a22ffd640 1 --2- 172.21.15.129:0/3391094333 >> v2:172.21.15.157:3300/0 conn(0x7f3a240ceb00 0x7f3a2415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.808+0000 7f3a23fff640 1 -- 172.21.15.129:0/3391094333 >> v2:172.21.15.157:3300/0 conn(0x7f3a240ceb00 msgr2=0x7f3a2415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.808+0000 7f3a23fff640 1 --2- 172.21.15.129:0/3391094333 >> v2:172.21.15.157:3300/0 conn(0x7f3a240ceb00 0x7f3a2415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:21.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.808+0000 7f3a23fff640 1 -- 172.21.15.129:0/3391094333 >> v1:172.21.15.157:6789/0 conn(0x7f3a240ccd60 legacy=0x7f3a2406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:21.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.808+0000 7f3a23fff640 1 -- 172.21.15.129:0/3391094333 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a240cd950 con 0x7f3a240cef40 2024-03-23T15:58:21.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.811+0000 7f3a23fff640 1 --2- 172.21.15.129:0/3391094333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a2415b8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009d90 tx=0x7f3a1c02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ad6f231c011f5a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.816+0000 7f3a227fc640 1 -- 172.21.15.129:0/3391094333 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c02cce0 con 0x7f3a240cef40 2024-03-23T15:58:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.816+0000 7f3a227fc640 1 -- 172.21.15.129:0/3391094333 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1c02ce40 con 0x7f3a240cef40 2024-03-23T15:58:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.816+0000 7f3a227fc640 1 -- 172.21.15.129:0/3391094333 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c034660 con 0x7f3a240cef40 2024-03-23T15:58:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 -- 172.21.15.129:0/3391094333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 msgr2=0x7f3a2415b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/3391094333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a2415b8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009d90 tx=0x7f3a1c02bf30 comp rx=0 tx=0).stop 2024-03-23T15:58:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 -- 172.21.15.129:0/3391094333 shutdown_connections 2024-03-23T15:58:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/3391094333 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a2415b8f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/3391094333 >> v2:172.21.15.157:3300/0 conn(0x7f3a240ceb00 0x7f3a2415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 -- 172.21.15.129:0/3391094333 >> 172.21.15.129:0/3391094333 conn(0x7f3a240b1840 msgr2=0x7f3a240b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:21.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 -- 172.21.15.129:0/3391094333 shutdown_connections 2024-03-23T15:58:21.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.817+0000 7f3a29e9a640 1 -- 172.21.15.129:0/3391094333 wait complete. 2024-03-23T15:58:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.819+0000 7f3a29e9a640 1 Processor -- start 2024-03-23T15:58:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.819+0000 7f3a29e9a640 1 -- start start 2024-03-23T15:58:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a29e9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a240ccd60 0x7f3a241f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a29e9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240ceb00 0x7f3a240c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a29e9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a24151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a29e9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241f1a80 con 0x7f3a240ccd60 2024-03-23T15:58:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a29e9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241f0c80 con 0x7f3a240ceb00 2024-03-23T15:58:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a29e9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a240c8520 con 0x7f3a240cef40 2024-03-23T15:58:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.820+0000 7f3a23fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a24151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.821+0000 7f3a237fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a240ccd60 0x7f3a241f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.821+0000 7f3a22ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240ceb00 0x7f3a240c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.821+0000 7f3a22ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240ceb00 0x7f3a240c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:34834/0 (socket says 172.21.15.129:34834) 2024-03-23T15:58:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.821+0000 7f3a22ffd640 1 -- 172.21.15.129:0/4009570953 learned_addr learned my addr 172.21.15.129:0/4009570953 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.824+0000 7f3a237fe640 1 -- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 msgr2=0x7f3a24151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.824+0000 7f3a237fe640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a24151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.824+0000 7f3a237fe640 1 -- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240ceb00 msgr2=0x7f3a240c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.824+0000 7f3a237fe640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240ceb00 0x7f3a240c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.824+0000 7f3a237fe640 1 -- 172.21.15.129:0/4009570953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c009a70 con 0x7f3a240ccd60 2024-03-23T15:58:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.827+0000 7f3a237fe640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a240ccd60 0x7f3a241f1540 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400c440 tx=0x7f3a14001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.831+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14005ce0 con 0x7f3a240ccd60 2024-03-23T15:58:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.831+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a14005e40 con 0x7f3a240ccd60 2024-03-23T15:58:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.831+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a240c8800 con 0x7f3a240ccd60 2024-03-23T15:58:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.831+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a140072d0 con 0x7f3a240ccd60 2024-03-23T15:58:21.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.831+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a240c8dc0 con 0x7f3a240ccd60 2024-03-23T15:58:21.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:21.832+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2404f530 con 0x7f3a240ccd60 2024-03-23T15:58:22.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.095+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3a140074f0 con 0x7f3a240ccd60 2024-03-23T15:58:22.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.096+0000 7f3a1bfff640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a0406b860 0x7f3a0406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:22.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.096+0000 7f3a22ffd640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a0406b860 0x7f3a0406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:22.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.097+0000 7f3a22ffd640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a0406b860 0x7f3a0406dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3a24068900 tx=0x7f3a0c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:22.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.119+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3a14058240 con 0x7f3a240ccd60 2024-03-23T15:58:22.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.120+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3a140544e0 con 0x7f3a240ccd60 2024-03-23T15:58:22.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.336+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a2415b290 con 0x7f3a240ccd60 2024-03-23T15:58:22.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.351+0000 7f3a1bfff640 1 -- 172.21.15.129:0/4009570953 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3a2415b290 con 0x7f3a240ccd60 2024-03-23T15:58:22.353 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:22.353 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:22.354 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a0406b860 msgr2=0x7f3a0406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a0406b860 0x7f3a0406dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3a24068900 tx=0x7f3a0c009190 comp rx=0 tx=0).stop 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a240ccd60 msgr2=0x7f3a241f1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a240ccd60 0x7f3a241f1540 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400c440 tx=0x7f3a14001e70 comp rx=0 tx=0).stop 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 shutdown_connections 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240cef40 0x7f3a24151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a0406b860 0x7f3a0406dd20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240ceb00 0x7f3a240c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 --2- 172.21.15.129:0/4009570953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a240ccd60 0x7f3a241f1540 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 >> 172.21.15.129:0/4009570953 conn(0x7f3a240b1840 msgr2=0x7f3a240b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 shutdown_connections 2024-03-23T15:58:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:22.353+0000 7f3a29e9a640 1 -- 172.21.15.129:0/4009570953 wait complete. 2024-03-23T15:58:22.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:22.975 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:23.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 Processor -- start 2024-03-23T15:58:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 -- start start 2024-03-23T15:58:24.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3ee006c030 0x7f3ee006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee00cbc60 0x7f3ee015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ee00cc080 0x7f3ee015de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3ee006cae0 con 0x7f3ee006c030 2024-03-23T15:58:24.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3ee00cabc0 con 0x7f3ee00cc080 2024-03-23T15:58:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.077+0000 7f3ee606f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee00caea0 con 0x7f3ee00cbc60 2024-03-23T15:58:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.078+0000 7f3edf7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3ee006c030 0x7f3ee006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.078+0000 7f3edffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ee00cc080 0x7f3ee015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.078+0000 7f3edeffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee00cbc60 0x7f3ee015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.078+0000 7f3edf7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3ee006c030 0x7f3ee006c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54836/0 (socket says 172.21.15.129:54836) 2024-03-23T15:58:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.078+0000 7f3edf7fe640 1 -- 172.21.15.129:0/362115931 learned_addr learned my addr 172.21.15.129:0/362115931 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.081+0000 7f3edf7fe640 1 -- 172.21.15.129:0/362115931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee00cbc60 msgr2=0x7f3ee015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.081+0000 7f3edf7fe640 1 --2- 172.21.15.129:0/362115931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee00cbc60 0x7f3ee015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.081+0000 7f3edf7fe640 1 -- 172.21.15.129:0/362115931 >> v2:172.21.15.157:3300/0 conn(0x7f3ee00cc080 msgr2=0x7f3ee015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.081+0000 7f3edf7fe640 1 --2- 172.21.15.129:0/362115931 >> v2:172.21.15.157:3300/0 conn(0x7f3ee00cc080 0x7f3ee015de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.081+0000 7f3edf7fe640 1 -- 172.21.15.129:0/362115931 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee00cb180 con 0x7f3ee006c030 2024-03-23T15:58:24.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.083+0000 7f3edf7fe640 1 --2- 172.21.15.129:0/362115931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee006c030 0x7f3ee006c410 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00c6b80 tx=0x7f3ed002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a442d11bad8f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:24.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.086+0000 7f3ede7fc640 1 -- 172.21.15.129:0/362115931 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00317e0 con 0x7f3ee006c030 2024-03-23T15:58:24.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.086+0000 7f3ede7fc640 1 -- 172.21.15.129:0/362115931 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed0031940 con 0x7f3ee006c030 2024-03-23T15:58:24.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.086+0000 7f3ede7fc640 1 -- 172.21.15.129:0/362115931 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0031cd0 con 0x7f3ee006c030 2024-03-23T15:58:24.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 -- 172.21.15.129:0/362115931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee006c030 msgr2=0x7f3ee006c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:24.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 --2- 172.21.15.129:0/362115931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee006c030 0x7f3ee006c410 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00c6b80 tx=0x7f3ed002b880 comp rx=0 tx=0).stop 2024-03-23T15:58:24.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 -- 172.21.15.129:0/362115931 shutdown_connections 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 --2- 172.21.15.129:0/362115931 >> v2:172.21.15.157:3300/0 conn(0x7f3ee00cc080 0x7f3ee015de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 --2- 172.21.15.129:0/362115931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee00cbc60 0x7f3ee015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 --2- 172.21.15.129:0/362115931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee006c030 0x7f3ee006c410 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 -- 172.21.15.129:0/362115931 >> 172.21.15.129:0/362115931 conn(0x7f3ee00cfb20 msgr2=0x7f3ee00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 -- 172.21.15.129:0/362115931 shutdown_connections 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.087+0000 7f3ee606f640 1 -- 172.21.15.129:0/362115931 wait complete. 2024-03-23T15:58:24.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.088+0000 7f3ee606f640 1 Processor -- start 2024-03-23T15:58:24.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 -- start start 2024-03-23T15:58:24.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee006c030 0x7f3ee01f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ee00cbc60 0x7f3ee01569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 0x7f3ee01f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee01f4100 con 0x7f3ee00cc080 2024-03-23T15:58:24.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee01effe0 con 0x7f3ee00cbc60 2024-03-23T15:58:24.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.089+0000 7f3ee606f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee015bce0 con 0x7f3ee006c030 2024-03-23T15:58:24.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.090+0000 7f3edffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 0x7f3ee01f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.090+0000 7f3edeffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ee00cbc60 0x7f3ee01569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.090+0000 7f3edf7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee006c030 0x7f3ee01f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.090+0000 7f3edffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 0x7f3ee01f0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54838/0 (socket says 172.21.15.129:54838) 2024-03-23T15:58:24.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.090+0000 7f3edffff640 1 -- 172.21.15.129:0/293635996 learned_addr learned my addr 172.21.15.129:0/293635996 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:24.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.094+0000 7f3edffff640 1 -- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee006c030 msgr2=0x7f3ee01f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:24.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.094+0000 7f3edffff640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee006c030 0x7f3ee01f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.094+0000 7f3edffff640 1 -- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ee00cbc60 msgr2=0x7f3ee01569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:24.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.094+0000 7f3edffff640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ee00cbc60 0x7f3ee01569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.094+0000 7f3edffff640 1 -- 172.21.15.129:0/293635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed00094d0 con 0x7f3ee00cc080 2024-03-23T15:58:24.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.096+0000 7f3edffff640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 0x7f3ee01f0470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800efc0 tx=0x7f3ed800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:24.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.101+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8014480 con 0x7f3ee00cc080 2024-03-23T15:58:24.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.101+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed800f040 con 0x7f3ee00cc080 2024-03-23T15:58:24.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.101+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8014820 con 0x7f3ee00cc080 2024-03-23T15:58:24.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.101+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee015a6a0 con 0x7f3ee00cc080 2024-03-23T15:58:24.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.101+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee015ac00 con 0x7f3ee00cc080 2024-03-23T15:58:24.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.102+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee0075740 con 0x7f3ee00cc080 2024-03-23T15:58:24.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.125+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ed8014980 con 0x7f3ee00cc080 2024-03-23T15:58:24.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.126+0000 7f3ec7fff640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ecc06b840 0x7f3ecc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:24.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.126+0000 7f3edf7fe640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ecc06b840 0x7f3ecc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:24.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.126+0000 7f3edf7fe640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ecc06b840 0x7f3ecc06dd00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0036000 tx=0x7f3ed003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.400+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3ed8059f90 con 0x7f3ee00cc080 2024-03-23T15:58:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.401+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ed8056230 con 0x7f3ee00cc080 2024-03-23T15:58:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.623+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ee01f0120 con 0x7f3ee00cc080 2024-03-23T15:58:24.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.637+0000 7f3ec7fff640 1 -- 172.21.15.129:0/293635996 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3ed8018030 con 0x7f3ee00cc080 2024-03-23T15:58:24.639 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:24.639 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:24.639 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ecc06b840 msgr2=0x7f3ecc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ecc06b840 0x7f3ecc06dd00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0036000 tx=0x7f3ed003d040 comp rx=0 tx=0).stop 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 msgr2=0x7f3ee01f0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 0x7f3ee01f0470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800efc0 tx=0x7f3ed800c490 comp rx=0 tx=0).stop 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 shutdown_connections 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ee00cc080 0x7f3ee01f0470 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3ecc06b840 0x7f3ecc06dd00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ee00cbc60 0x7f3ee01569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 --2- 172.21.15.129:0/293635996 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ee006c030 0x7f3ee01f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 >> 172.21.15.129:0/293635996 conn(0x7f3ee00cfb20 msgr2=0x7f3ee00ca100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:24.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 shutdown_connections 2024-03-23T15:58:24.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:24.639+0000 7f3ee606f640 1 -- 172.21.15.129:0/293635996 wait complete. 2024-03-23T15:58:25.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:25.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:26.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:26.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.358+0000 7f17779b6640 1 Processor -- start 2024-03-23T15:58:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.358+0000 7f17779b6640 1 -- start start 2024-03-23T15:58:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.359+0000 7f17779b6640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f177006cf70 0x7f17700cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.359+0000 7f17779b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f17701540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.359+0000 7f17779b6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1770070640 con 0x7f177015dba0 2024-03-23T15:58:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.359+0000 7f17779b6640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f17700c4c00 con 0x7f177006cf70 2024-03-23T15:58:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.359+0000 7f17779b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f177006c240 con 0x7f177015df80 2024-03-23T15:58:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.360+0000 7f1774f51640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f177006cf70 0x7f17700cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.360+0000 7f1775f53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f17701540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.360+0000 7f1775752640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f177015dba0 0x7f177006ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44370/0 (socket says 172.21.15.129:44370) 2024-03-23T15:58:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.360+0000 7f1775752640 1 -- 172.21.15.129:0/1506553222 learned_addr learned my addr 172.21.15.129:0/1506553222 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:26.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.363+0000 7f1774f51640 1 -- 172.21.15.129:0/1506553222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 msgr2=0x7f17701540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:26.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.363+0000 7f1774f51640 1 --2- 172.21.15.129:0/1506553222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f17701540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.363+0000 7f1774f51640 1 -- 172.21.15.129:0/1506553222 >> v1:172.21.15.157:6789/0 conn(0x7f177015dba0 legacy=0x7f177006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:26.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.363+0000 7f1774f51640 1 -- 172.21.15.129:0/1506553222 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17700cbb20 con 0x7f177006cf70 2024-03-23T15:58:26.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.365+0000 7f1774f51640 1 --2- 172.21.15.129:0/1506553222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17700cca30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f17600097c0 tx=0x7f176002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb9f3845f5ff131 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.368+0000 7f1767fff640 1 -- 172.21.15.129:0/1506553222 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760031890 con 0x7f177006cf70 2024-03-23T15:58:26.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.370+0000 7f1767fff640 1 -- 172.21.15.129:0/1506553222 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17600319f0 con 0x7f177006cf70 2024-03-23T15:58:26.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.370+0000 7f1767fff640 1 -- 172.21.15.129:0/1506553222 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760031d80 con 0x7f177006cf70 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 -- 172.21.15.129:0/1506553222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 msgr2=0x7f17700cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 --2- 172.21.15.129:0/1506553222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17700cca30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f17600097c0 tx=0x7f176002b930 comp rx=0 tx=0).stop 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 -- 172.21.15.129:0/1506553222 shutdown_connections 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 --2- 172.21.15.129:0/1506553222 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f17701540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 --2- 172.21.15.129:0/1506553222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17700cca30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 -- 172.21.15.129:0/1506553222 >> 172.21.15.129:0/1506553222 conn(0x7f1770071300 msgr2=0x7f17700716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:26.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 -- 172.21.15.129:0/1506553222 shutdown_connections 2024-03-23T15:58:26.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.371+0000 7f17779b6640 1 -- 172.21.15.129:0/1506553222 wait complete. 2024-03-23T15:58:26.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.372+0000 7f17779b6640 1 Processor -- start 2024-03-23T15:58:26.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.372+0000 7f17779b6640 1 -- start start 2024-03-23T15:58:26.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.373+0000 7f17779b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17701447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:26.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.373+0000 7f17779b6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f177015dba0 0x7f17701f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:26.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.373+0000 7f17779b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f177014df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:26.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.373+0000 7f17779b6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1770155e40 con 0x7f177006cf70 2024-03-23T15:58:26.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.373+0000 7f17779b6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f177014be60 con 0x7f177015dba0 2024-03-23T15:58:26.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.373+0000 7f17779b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17701efde0 con 0x7f177015df80 2024-03-23T15:58:26.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.374+0000 7f1775f53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f177014df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:26.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.374+0000 7f1775752640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17701447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:26.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.374+0000 7f1774f51640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f177015dba0 0x7f17701f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:26.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.374+0000 7f1775752640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17701447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54852/0 (socket says 172.21.15.129:54852) 2024-03-23T15:58:26.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.374+0000 7f1775752640 1 -- 172.21.15.129:0/2008431532 learned_addr learned my addr 172.21.15.129:0/2008431532 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:26.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.377+0000 7f1775752640 1 -- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 msgr2=0x7f177014df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:26.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.377+0000 7f1775752640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f177014df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.377+0000 7f1775752640 1 -- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f177015dba0 msgr2=0x7f17701f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:26.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.377+0000 7f1775752640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f177015dba0 0x7f17701f04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.377+0000 7f1775752640 1 -- 172.21.15.129:0/2008431532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17600093a0 con 0x7f177006cf70 2024-03-23T15:58:26.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.380+0000 7f1775752640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17701447f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f176c00c430 tx=0x7f176c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.384+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c005ce0 con 0x7f177006cf70 2024-03-23T15:58:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.384+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176c005e40 con 0x7f177006cf70 2024-03-23T15:58:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.384+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17701f0060 con 0x7f177006cf70 2024-03-23T15:58:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.384+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c0072f0 con 0x7f177006cf70 2024-03-23T15:58:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.384+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1770144d30 con 0x7f177006cf70 2024-03-23T15:58:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.385+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17700686f0 con 0x7f177006cf70 2024-03-23T15:58:26.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.409+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f176c007450 con 0x7f177006cf70 2024-03-23T15:58:26.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.409+0000 7f1765ffb640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f175406b860 0x7f175406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:26.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.409+0000 7f1774f51640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f175406b860 0x7f175406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.410+0000 7f1774f51640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f175406b860 0x7f175406dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1760036000 tx=0x7f1760008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:26.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.684+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f176c0583c0 con 0x7f177006cf70 2024-03-23T15:58:26.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.684+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f176c054660 con 0x7f177006cf70 2024-03-23T15:58:26.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.932+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1770075720 con 0x7f177006cf70 2024-03-23T15:58:26.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.944+0000 7f1765ffb640 1 -- 172.21.15.129:0/2008431532 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f176c017020 con 0x7f177006cf70 2024-03-23T15:58:26.947 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:26.947 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:26.947 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:26.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f175406b860 msgr2=0x7f175406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:26.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f175406b860 0x7f175406dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1760036000 tx=0x7f1760008030 comp rx=0 tx=0).stop 2024-03-23T15:58:26.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 msgr2=0x7f17701447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:26.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17701447f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f176c00c430 tx=0x7f176c00c8f0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 shutdown_connections 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f177015df80 0x7f177014df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f175406b860 0x7f175406dd20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f177015dba0 0x7f17701f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 --2- 172.21.15.129:0/2008431532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f177006cf70 0x7f17701447f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 >> 172.21.15.129:0/2008431532 conn(0x7f1770071300 msgr2=0x7f17700cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 shutdown_connections 2024-03-23T15:58:26.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:26.947+0000 7f17779b6640 1 -- 172.21.15.129:0/2008431532 wait complete. 2024-03-23T15:58:27.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:27.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:28.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.667+0000 7fc08f513640 1 Processor -- start 2024-03-23T15:58:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.668+0000 7fc08f513640 1 -- start start 2024-03-23T15:58:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.668+0000 7fc08f513640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc08815dc60 0x7fc08806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.668+0000 7fc08f513640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc08806f570 0x7fc0880cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.668+0000 7fc08f513640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc08806d2c0 con 0x7fc08815e040 2024-03-23T15:58:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.668+0000 7fc08f513640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc08815e740 con 0x7fc08815dc60 2024-03-23T15:58:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.668+0000 7fc08f513640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc0880ccd00 con 0x7fc08806f570 2024-03-23T15:58:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.669+0000 7fc08d2af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc08815dc60 0x7fc08806cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.669+0000 7fc08dab0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc08806f570 0x7fc0880cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:28.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.669+0000 7fc08d2af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc08815dc60 0x7fc08806cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54866/0 (socket says 172.21.15.129:54866) 2024-03-23T15:58:28.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.669+0000 7fc08d2af640 1 -- 172.21.15.129:0/3800921670 learned_addr learned my addr 172.21.15.129:0/3800921670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:28.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.673+0000 7fc08dab0640 1 -- 172.21.15.129:0/3800921670 >> v2:172.21.15.129:3300/0 conn(0x7fc08815dc60 msgr2=0x7fc08806cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:28.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.673+0000 7fc08dab0640 1 --2- 172.21.15.129:0/3800921670 >> v2:172.21.15.129:3300/0 conn(0x7fc08815dc60 0x7fc08806cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:28.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.673+0000 7fc08dab0640 1 -- 172.21.15.129:0/3800921670 >> v1:172.21.15.129:6789/0 conn(0x7fc08815e040 legacy=0x7fc0880cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:28.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.673+0000 7fc08dab0640 1 -- 172.21.15.129:0/3800921670 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0880ccfe0 con 0x7fc08806f570 2024-03-23T15:58:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.675+0000 7fc08dab0640 1 --2- 172.21.15.129:0/3800921670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08806f570 0x7fc0880cb7c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc08815b410 tx=0x7fc08002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd99fb541a21b39a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.680+0000 7fc07ffff640 1 -- 172.21.15.129:0/3800921670 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08002cbf0 con 0x7fc08806f570 2024-03-23T15:58:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.680+0000 7fc07ffff640 1 -- 172.21.15.129:0/3800921670 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08002cd50 con 0x7fc08806f570 2024-03-23T15:58:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.680+0000 7fc07ffff640 1 -- 172.21.15.129:0/3800921670 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080034660 con 0x7fc08806f570 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.680+0000 7fc08f513640 1 -- 172.21.15.129:0/3800921670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08806f570 msgr2=0x7fc0880cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.680+0000 7fc08f513640 1 --2- 172.21.15.129:0/3800921670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08806f570 0x7fc0880cb7c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc08815b410 tx=0x7fc08002be40 comp rx=0 tx=0).stop 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.681+0000 7fc08f513640 1 -- 172.21.15.129:0/3800921670 shutdown_connections 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.681+0000 7fc08f513640 1 --2- 172.21.15.129:0/3800921670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08806f570 0x7fc0880cb7c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.681+0000 7fc08f513640 1 --2- 172.21.15.129:0/3800921670 >> v2:172.21.15.129:3300/0 conn(0x7fc08815dc60 0x7fc08806cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.681+0000 7fc08f513640 1 -- 172.21.15.129:0/3800921670 >> 172.21.15.129:0/3800921670 conn(0x7fc088071300 msgr2=0x7fc0880716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.681+0000 7fc08f513640 1 -- 172.21.15.129:0/3800921670 shutdown_connections 2024-03-23T15:58:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.681+0000 7fc08f513640 1 -- 172.21.15.129:0/3800921670 wait complete. 2024-03-23T15:58:28.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.682+0000 7fc08f513640 1 Processor -- start 2024-03-23T15:58:28.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.682+0000 7fc08f513640 1 -- start start 2024-03-23T15:58:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08f513640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc08806f570 0x7fc0880b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08f513640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 0x7fc0880b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08f513640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08815e040 0x7fc0880aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08f513640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0881e5980 con 0x7fc08815dc60 2024-03-23T15:58:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08f513640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc0881e6220 con 0x7fc08815e040 2024-03-23T15:58:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08f513640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc0880b1fb0 con 0x7fc08806f570 2024-03-23T15:58:28.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08caae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 0x7fc0880b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:28.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08dab0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08815e040 0x7fc0880aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:28.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08d2af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc08806f570 0x7fc0880b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:28.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08caae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 0x7fc0880b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54880/0 (socket says 172.21.15.129:54880) 2024-03-23T15:58:28.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.683+0000 7fc08caae640 1 -- 172.21.15.129:0/317069524 learned_addr learned my addr 172.21.15.129:0/317069524 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:28.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.687+0000 7fc08caae640 1 -- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc08806f570 msgr2=0x7fc0880b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:28.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.687+0000 7fc08caae640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc08806f570 0x7fc0880b1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:28.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.687+0000 7fc08caae640 1 -- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08815e040 msgr2=0x7fc0880aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:28.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.687+0000 7fc08caae640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08815e040 0x7fc0880aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:28.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.687+0000 7fc08caae640 1 -- 172.21.15.129:0/317069524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc080009a70 con 0x7fc08815dc60 2024-03-23T15:58:28.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.692+0000 7fc08caae640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 0x7fc0880b19d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc07800c470 tx=0x7fc078001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.697+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0780052d0 con 0x7fc08815dc60 2024-03-23T15:58:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.697+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc078005430 con 0x7fc08815dc60 2024-03-23T15:58:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.697+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0880b2290 con 0x7fc08815dc60 2024-03-23T15:58:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.697+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc078007260 con 0x7fc08815dc60 2024-03-23T15:58:28.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.698+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0881e5eb0 con 0x7fc08815dc60 2024-03-23T15:58:28.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.699+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0881e6040 con 0x7fc08815dc60 2024-03-23T15:58:28.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.978+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc0780073c0 con 0x7fc08815dc60 2024-03-23T15:58:28.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.978+0000 7fc07dffb640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc06c06b810 0x7fc06c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:28.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.979+0000 7fc08d2af640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc06c06b810 0x7fc06c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:28.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:28.979+0000 7fc08d2af640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc06c06b810 0x7fc06c06dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc084009b30 tx=0x7fc084009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:29.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.003+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc078007670 con 0x7fc08815dc60 2024-03-23T15:58:29.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.003+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc0780544b0 con 0x7fc08815dc60 2024-03-23T15:58:29.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.227+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc08806db00 con 0x7fc08815dc60 2024-03-23T15:58:29.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.259+0000 7fc07dffb640 1 -- 172.21.15.129:0/317069524 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc078020020 con 0x7fc08815dc60 2024-03-23T15:58:29.262 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:29.262 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:29.263 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:29.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc06c06b810 msgr2=0x7fc06c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:29.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc06c06b810 0x7fc06c06dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc084009b30 tx=0x7fc084009240 comp rx=0 tx=0).stop 2024-03-23T15:58:29.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 msgr2=0x7fc0880b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:29.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 0x7fc0880b19d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc07800c470 tx=0x7fc078001e70 comp rx=0 tx=0).stop 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 shutdown_connections 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc06c06b810 0x7fc06c06dcd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc08815e040 0x7fc0880aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc08815dc60 0x7fc0880b19d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 --2- 172.21.15.129:0/317069524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc08806f570 0x7fc0880b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 >> 172.21.15.129:0/317069524 conn(0x7fc088071300 msgr2=0x7fc088158910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 shutdown_connections 2024-03-23T15:58:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:29.262+0000 7fc08f513640 1 -- 172.21.15.129:0/317069524 wait complete. 2024-03-23T15:58:29.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:29.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:30.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:30.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.983+0000 7f6259c3b640 1 Processor -- start 2024-03-23T15:58:30.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.983+0000 7f6259c3b640 1 -- start start 2024-03-23T15:58:30.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6259c3b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 0x7f625415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:30.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6259c3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f625406c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6259c3b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f62540cef40 0x7f625406c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6259c3b640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f625415b290 con 0x7f62540cef40 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6259c3b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f62540ce800 con 0x7f62540ccd60 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6259c3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6254154a70 con 0x7f62540ceb00 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f62537fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 0x7f625415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6253fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f62540cef40 0x7f625406c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.984+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f625406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.985+0000 7f62537fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 0x7f625415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:49966/0 (socket says 172.21.15.129:49966) 2024-03-23T15:58:30.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.985+0000 7f62537fe640 1 -- 172.21.15.129:0/89617423 learned_addr learned my addr 172.21.15.129:0/89617423 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:30.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.988+0000 7f6253fff640 1 -- 172.21.15.129:0/89617423 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 msgr2=0x7f625406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:30.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.988+0000 7f6253fff640 1 --2- 172.21.15.129:0/89617423 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f625406c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:30.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.988+0000 7f6253fff640 1 -- 172.21.15.129:0/89617423 >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 msgr2=0x7f625415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:30.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.988+0000 7f6253fff640 1 --2- 172.21.15.129:0/89617423 >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 0x7f625415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:30.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.988+0000 7f6253fff640 1 -- 172.21.15.129:0/89617423 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62540cd950 con 0x7f62540cef40 2024-03-23T15:58:30.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.988+0000 7f62537fe640 1 --2- 172.21.15.129:0/89617423 >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 0x7f625415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:58:30.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.991+0000 7f6253fff640 1 --2- 172.21.15.129:0/89617423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540cef40 0x7f625406c7b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f625408cf40 tx=0x7f624c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88d8fabc9735637 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:30.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.995+0000 7f62527fc640 1 -- 172.21.15.129:0/89617423 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c02cc60 con 0x7f62540cef40 2024-03-23T15:58:30.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.995+0000 7f62527fc640 1 -- 172.21.15.129:0/89617423 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f624c02cdc0 con 0x7f62540cef40 2024-03-23T15:58:30.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.995+0000 7f62527fc640 1 -- 172.21.15.129:0/89617423 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c034660 con 0x7f62540cef40 2024-03-23T15:58:30.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 -- 172.21.15.129:0/89617423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540cef40 msgr2=0x7f625406c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:30.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 --2- 172.21.15.129:0/89617423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540cef40 0x7f625406c7b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f625408cf40 tx=0x7f624c02beb0 comp rx=0 tx=0).stop 2024-03-23T15:58:30.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 -- 172.21.15.129:0/89617423 shutdown_connections 2024-03-23T15:58:30.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 --2- 172.21.15.129:0/89617423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540cef40 0x7f625406c7b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:30.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 --2- 172.21.15.129:0/89617423 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f625406c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:30.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 --2- 172.21.15.129:0/89617423 >> v2:172.21.15.157:3300/0 conn(0x7f62540ccd60 0x7f625415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:30.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 -- 172.21.15.129:0/89617423 >> 172.21.15.129:0/89617423 conn(0x7f62540b1840 msgr2=0x7f62540b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:30.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 -- 172.21.15.129:0/89617423 shutdown_connections 2024-03-23T15:58:30.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.996+0000 7f6259c3b640 1 -- 172.21.15.129:0/89617423 wait complete. 2024-03-23T15:58:30.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.997+0000 7f6259c3b640 1 Processor -- start 2024-03-23T15:58:31.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 -- start start 2024-03-23T15:58:31.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 0x7f62541e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:31.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f6254150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:31.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62540cef40 0x7f62541e5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:31.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62541e5170 con 0x7f62540ccd60 2024-03-23T15:58:31.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62541e4e70 con 0x7f62540cef40 2024-03-23T15:58:31.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.998+0000 7f6259c3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f625415ce50 con 0x7f62540ceb00 2024-03-23T15:58:31.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.999+0000 7f62537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 0x7f62541e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:31.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.999+0000 7f6253fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62540cef40 0x7f62541e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:31.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.999+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f6254150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:31.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.999+0000 7f62537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 0x7f62541e4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44610/0 (socket says 172.21.15.129:44610) 2024-03-23T15:58:31.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:30.999+0000 7f62537fe640 1 -- 172.21.15.129:0/3887317610 learned_addr learned my addr 172.21.15.129:0/3887317610 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:31.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.002+0000 7f62537fe640 1 -- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 msgr2=0x7f6254150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:31.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.002+0000 7f62537fe640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f6254150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.002+0000 7f62537fe640 1 -- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62540cef40 msgr2=0x7f62541e5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:31.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.002+0000 7f62537fe640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62540cef40 0x7f62541e5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.002+0000 7f62537fe640 1 -- 172.21.15.129:0/3887317610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440094d0 con 0x7f62540ccd60 2024-03-23T15:58:31.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.005+0000 7f62537fe640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 0x7f62541e4420 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f624400e9e0 tx=0x7f624400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:31.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.009+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624400c9d0 con 0x7f62540ccd60 2024-03-23T15:58:31.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.009+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f624400cb30 con 0x7f62540ccd60 2024-03-23T15:58:31.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.009+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f624c009a70 con 0x7f62540ccd60 2024-03-23T15:58:31.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.009+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624400cea0 con 0x7f62540ccd60 2024-03-23T15:58:31.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.009+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f625415d410 con 0x7f62540ccd60 2024-03-23T15:58:31.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.010+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f625404f530 con 0x7f62540ccd60 2024-03-23T15:58:31.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.263+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6244013470 con 0x7f62540ccd60 2024-03-23T15:58:31.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.263+0000 7f623bfff640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f624006b840 0x7f624006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:31.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.263+0000 7f6252ffd640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f624006b840 0x7f624006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.264+0000 7f6252ffd640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f624006b840 0x7f624006dd00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f625415e360 tx=0x7f6248009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:31.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.279+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6244014e40 con 0x7f62540ccd60 2024-03-23T15:58:31.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.279+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6244055700 con 0x7f62540ccd60 2024-03-23T15:58:31.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.519+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f625406db50 con 0x7f62540ccd60 2024-03-23T15:58:31.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.532+0000 7f623bfff640 1 -- 172.21.15.129:0/3887317610 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6244017090 con 0x7f62540ccd60 2024-03-23T15:58:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:31.535 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:31.535 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:31.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f624006b840 msgr2=0x7f624006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:31.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f624006b840 0x7f624006dd00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f625415e360 tx=0x7f6248009190 comp rx=0 tx=0).stop 2024-03-23T15:58:31.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 msgr2=0x7f62541e4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:31.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 0x7f62541e4420 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f624400e9e0 tx=0x7f624400eea0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 shutdown_connections 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f624006b840 0x7f624006dd00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62540cef40 0x7f62541e5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62540ceb00 0x7f6254150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 --2- 172.21.15.129:0/3887317610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62540ccd60 0x7f62541e4420 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 >> 172.21.15.129:0/3887317610 conn(0x7f62540b1840 msgr2=0x7f62541e6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 shutdown_connections 2024-03-23T15:58:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:31.535+0000 7f6259c3b640 1 -- 172.21.15.129:0/3887317610 wait complete. 2024-03-23T15:58:32.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:32.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:33.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:33.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.253+0000 7f8ad7dfd640 1 Processor -- start 2024-03-23T15:58:33.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.254+0000 7f8ad7dfd640 1 -- start start 2024-03-23T15:58:33.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.255+0000 7f8ad7dfd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad00cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:33.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.255+0000 7f8ad7dfd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ad015be30 0x7f8ad00c33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:33.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.255+0000 7f8ad7dfd640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad015c5a0 con 0x7f8ad015ba50 2024-03-23T15:58:33.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.255+0000 7f8ad7dfd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad00705c0 con 0x7f8ad015be30 2024-03-23T15:58:33.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.255+0000 7f8ad7dfd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad006f680 con 0x7f8ad00709e0 2024-03-23T15:58:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.256+0000 7f8ad639a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8ad015be30 0x7f8ad00c33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.256+0000 7f8ad5398640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad00cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.256+0000 7f8ad5b99640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8ad015ba50 0x7f8ad00cc880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38192/0 (socket says 172.21.15.129:38192) 2024-03-23T15:58:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.256+0000 7f8ad5b99640 1 -- 172.21.15.129:0/3252399244 learned_addr learned my addr 172.21.15.129:0/3252399244 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.259+0000 7f8ad639a640 1 -- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 msgr2=0x7f8ad00cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.259+0000 7f8ad639a640 1 --2- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad00cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.259+0000 7f8ad639a640 1 -- 172.21.15.129:0/3252399244 >> v1:172.21.15.129:6789/0 conn(0x7f8ad015ba50 legacy=0x7f8ad00cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.259+0000 7f8ad639a640 1 -- 172.21.15.129:0/3252399244 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad006f940 con 0x7f8ad015be30 2024-03-23T15:58:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.261+0000 7f8ad639a640 1 --2- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015be30 0x7f8ad00c33d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009d90 tx=0x7f8ac802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f7c102d96255e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:33.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.263+0000 7f8ad4b97640 1 -- 172.21.15.129:0/3252399244 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac802cce0 con 0x7f8ad015be30 2024-03-23T15:58:33.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.266+0000 7f8ad4b97640 1 -- 172.21.15.129:0/3252399244 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac802ce40 con 0x7f8ad015be30 2024-03-23T15:58:33.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.266+0000 7f8ad4b97640 1 -- 172.21.15.129:0/3252399244 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8034660 con 0x7f8ad015be30 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015be30 msgr2=0x7f8ad00c33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015be30 0x7f8ad00c33d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009d90 tx=0x7f8ac802bf30 comp rx=0 tx=0).stop 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/3252399244 shutdown_connections 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015be30 0x7f8ad00c33d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/3252399244 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad00cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/3252399244 >> 172.21.15.129:0/3252399244 conn(0x7f8ad0071300 msgr2=0x7f8ad00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/3252399244 shutdown_connections 2024-03-23T15:58:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.267+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/3252399244 wait complete. 2024-03-23T15:58:33.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.268+0000 7f8ad7dfd640 1 Processor -- start 2024-03-23T15:58:33.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.268+0000 7f8ad7dfd640 1 -- start start 2024-03-23T15:58:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad7dfd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad7dfd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015ba50 0x7f8ad00c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad7dfd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ad015be30 0x7f8ad014e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad7dfd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad00c98c0 con 0x7f8ad015be30 2024-03-23T15:58:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad7dfd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad01efe60 con 0x7f8ad015ba50 2024-03-23T15:58:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad7dfd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad014d390 con 0x7f8ad00709e0 2024-03-23T15:58:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad639a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ad015be30 0x7f8ad014e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad5398640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015ba50 0x7f8ad00c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.269+0000 7f8ad5b99640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.271+0000 7f8ad5398640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015ba50 0x7f8ad00c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:50002/0 (socket says 172.21.15.129:50002) 2024-03-23T15:58:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.271+0000 7f8ad5398640 1 -- 172.21.15.129:0/768464024 learned_addr learned my addr 172.21.15.129:0/768464024 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:33.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.273+0000 7f8ad639a640 1 -- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 msgr2=0x7f8ad006ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:33.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.273+0000 7f8ad639a640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad006ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.273+0000 7f8ad639a640 1 -- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015ba50 msgr2=0x7f8ad00c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:33.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.273+0000 7f8ad639a640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015ba50 0x7f8ad00c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.273+0000 7f8ad639a640 1 -- 172.21.15.129:0/768464024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac8009a70 con 0x7f8ad015be30 2024-03-23T15:58:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.275+0000 7f8ad639a640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ad015be30 0x7f8ad014e860 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ad006c070 tx=0x7f8ac8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:33.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.279+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac803d600 con 0x7f8ad015be30 2024-03-23T15:58:33.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.279+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac803d780 con 0x7f8ad015be30 2024-03-23T15:58:33.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.279+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac803db30 con 0x7f8ad015be30 2024-03-23T15:58:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.281+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad00c8cf0 con 0x7f8ad015be30 2024-03-23T15:58:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.281+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad00c9270 con 0x7f8ad015be30 2024-03-23T15:58:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.282+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad00c9a30 con 0x7f8ad015be30 2024-03-23T15:58:33.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.312+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8ac803b040 con 0x7f8ad015be30 2024-03-23T15:58:33.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.313+0000 7f8ac67fc640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8ab406b860 0x7f8ab406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:33.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.313+0000 7f8ad5b99640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8ab406b860 0x7f8ab406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:33.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.314+0000 7f8ad5b99640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8ab406b860 0x7f8ab406dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8acc007fd0 tx=0x7f8acc007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:33.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.569+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8ac807af20 con 0x7f8ad015be30 2024-03-23T15:58:33.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.569+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8ac80771c0 con 0x7f8ad015be30 2024-03-23T15:58:33.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.792+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ad00c38e0 con 0x7f8ad015be30 2024-03-23T15:58:33.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.827+0000 7f8ac67fc640 1 -- 172.21.15.129:0/768464024 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8ac8039070 con 0x7f8ad015be30 2024-03-23T15:58:33.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:33.829 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:33.829 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.829+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8ab406b860 msgr2=0x7f8ab406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.829+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8ab406b860 0x7f8ab406dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8acc007fd0 tx=0x7f8acc007e80 comp rx=0 tx=0).stop 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.829+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ad015be30 msgr2=0x7f8ad014e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.829+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ad015be30 0x7f8ad014e860 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ad006c070 tx=0x7f8ac8004a40 comp rx=0 tx=0).stop 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 shutdown_connections 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ad015be30 0x7f8ad014e860 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8ab406b860 0x7f8ab406dd20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ad015ba50 0x7f8ad00c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 --2- 172.21.15.129:0/768464024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8ad00709e0 0x7f8ad006ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:33.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 >> 172.21.15.129:0/768464024 conn(0x7f8ad0071300 msgr2=0x7f8ad015a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:33.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 shutdown_connections 2024-03-23T15:58:33.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:33.830+0000 7f8ad7dfd640 1 -- 172.21.15.129:0/768464024 wait complete. 2024-03-23T15:58:34.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:34.447 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:35.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:35.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.550+0000 7f686d8d4640 1 Processor -- start 2024-03-23T15:58:35.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.550+0000 7f686d8d4640 1 -- start start 2024-03-23T15:58:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.551+0000 7f686d8d4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f68681544b0 0x7f686815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.551+0000 7f686d8d4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f68680c6000 0x7f68680c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.551+0000 7f686d8d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68680cbcb0 con 0x7f68680caa00 2024-03-23T15:58:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.551+0000 7f686d8d4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f68680c5be0 con 0x7f68681544b0 2024-03-23T15:58:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.551+0000 7f686d8d4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f68680c69b0 con 0x7f68680c6000 2024-03-23T15:58:35.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.552+0000 7f6866ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f68681544b0 0x7f686815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:35.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.553+0000 7f6866ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f68681544b0 0x7f686815a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44622/0 (socket says 172.21.15.129:44622) 2024-03-23T15:58:35.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.553+0000 7f6866ffd640 1 -- 172.21.15.129:0/3207166623 learned_addr learned my addr 172.21.15.129:0/3207166623 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:35.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.554+0000 7f68677fe640 1 --2- 172.21.15.129:0/3207166623 >> v2:172.21.15.157:3300/0 conn(0x7f68680c6000 0x7f68680c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.555+0000 7f6866ffd640 1 -- 172.21.15.129:0/3207166623 >> v2:172.21.15.157:3300/0 conn(0x7f68680c6000 msgr2=0x7f68680c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:35.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.555+0000 7f6866ffd640 1 --2- 172.21.15.129:0/3207166623 >> v2:172.21.15.157:3300/0 conn(0x7f68680c6000 0x7f68680c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:35.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.555+0000 7f6866ffd640 1 -- 172.21.15.129:0/3207166623 >> v1:172.21.15.157:6789/0 conn(0x7f68680caa00 legacy=0x7f686806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:35.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.555+0000 7f6866ffd640 1 -- 172.21.15.129:0/3207166623 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686815b280 con 0x7f68681544b0 2024-03-23T15:58:35.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.557+0000 7f6866ffd640 1 --2- 172.21.15.129:0/3207166623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68681544b0 0x7f686815a2e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f68680b3b50 tx=0x7f685802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc3160f04af62ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:35.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.559+0000 7f6865ffb640 1 -- 172.21.15.129:0/3207166623 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685802f850 con 0x7f68681544b0 2024-03-23T15:58:35.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.561+0000 7f6865ffb640 1 -- 172.21.15.129:0/3207166623 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685802f9b0 con 0x7f68681544b0 2024-03-23T15:58:35.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.561+0000 7f6865ffb640 1 -- 172.21.15.129:0/3207166623 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685802fd40 con 0x7f68681544b0 2024-03-23T15:58:35.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 -- 172.21.15.129:0/3207166623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68681544b0 msgr2=0x7f686815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:35.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 --2- 172.21.15.129:0/3207166623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68681544b0 0x7f686815a2e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f68680b3b50 tx=0x7f685802ba30 comp rx=0 tx=0).stop 2024-03-23T15:58:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 -- 172.21.15.129:0/3207166623 shutdown_connections 2024-03-23T15:58:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 --2- 172.21.15.129:0/3207166623 >> v2:172.21.15.157:3300/0 conn(0x7f68680c6000 0x7f68680c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 --2- 172.21.15.129:0/3207166623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68681544b0 0x7f686815a2e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 -- 172.21.15.129:0/3207166623 >> 172.21.15.129:0/3207166623 conn(0x7f6868061b50 msgr2=0x7f6868061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 -- 172.21.15.129:0/3207166623 shutdown_connections 2024-03-23T15:58:35.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.562+0000 7f686d8d4640 1 -- 172.21.15.129:0/3207166623 wait complete. 2024-03-23T15:58:35.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.563+0000 7f686d8d4640 1 Processor -- start 2024-03-23T15:58:35.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.563+0000 7f686d8d4640 1 -- start start 2024-03-23T15:58:35.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.564+0000 7f686d8d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f68680c6000 0x7f68681593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:35.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.564+0000 7f686d8d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 0x7f68680cf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:35.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.564+0000 7f686d8d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68681544b0 0x7f68681f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:35.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.564+0000 7f686d8d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68680c9720 con 0x7f68680caa00 2024-03-23T15:58:35.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.564+0000 7f686d8d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f68680c9ce0 con 0x7f68681544b0 2024-03-23T15:58:35.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.564+0000 7f686d8d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f686806c3e0 con 0x7f68680c6000 2024-03-23T15:58:35.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.565+0000 7f6866ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f68680c6000 0x7f68681593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:35.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.565+0000 7f68667fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 0x7f68680cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:35.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.565+0000 7f68667fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 0x7f68680cf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44628/0 (socket says 172.21.15.129:44628) 2024-03-23T15:58:35.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.565+0000 7f68667fc640 1 -- 172.21.15.129:0/2514685817 learned_addr learned my addr 172.21.15.129:0/2514685817 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:35.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.565+0000 7f68677fe640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68681544b0 0x7f68681f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:35.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.569+0000 7f68667fc640 1 -- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f68680c6000 msgr2=0x7f68681593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:35.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.569+0000 7f68667fc640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f68680c6000 0x7f68681593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:35.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.569+0000 7f68667fc640 1 -- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68681544b0 msgr2=0x7f68681f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:35.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.569+0000 7f68667fc640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68681544b0 0x7f68681f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:35.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.569+0000 7f68667fc640 1 -- 172.21.15.129:0/2514685817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68580094d0 con 0x7f68680caa00 2024-03-23T15:58:35.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.571+0000 7f68667fc640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 0x7f68680cf930 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f685c00c400 tx=0x7f685c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:35.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.575+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0052d0 con 0x7f68680caa00 2024-03-23T15:58:35.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.575+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685c005450 con 0x7f68680caa00 2024-03-23T15:58:35.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.575+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c007260 con 0x7f68680caa00 2024-03-23T15:58:35.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.575+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686815a590 con 0x7f68680caa00 2024-03-23T15:58:35.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.576+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68680c9970 con 0x7f68680caa00 2024-03-23T15:58:35.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.577+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68680b5300 con 0x7f68680caa00 2024-03-23T15:58:35.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.595+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f685c0073c0 con 0x7f68680caa00 2024-03-23T15:58:35.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.595+0000 7f684ffff640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f685406b810 0x7f685406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:35.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.596+0000 7f6866ffd640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f685406b810 0x7f685406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:35.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.596+0000 7f6866ffd640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f685406b810 0x7f685406dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6858036000 tx=0x7f6858008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:35.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.600+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f685c007670 con 0x7f68680caa00 2024-03-23T15:58:35.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:35.843+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f685c054980 con 0x7f68680caa00 2024-03-23T15:58:36.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.080+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6868075740 con 0x7f68680caa00 2024-03-23T15:58:36.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.095+0000 7f684ffff640 1 -- 172.21.15.129:0/2514685817 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f685c020020 con 0x7f68680caa00 2024-03-23T15:58:36.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:36.097 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:36.097 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:36.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f685406b810 msgr2=0x7f685406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:36.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f685406b810 0x7f685406dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6858036000 tx=0x7f6858008030 comp rx=0 tx=0).stop 2024-03-23T15:58:36.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 msgr2=0x7f68680cf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:36.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 0x7f68680cf930 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f685c00c400 tx=0x7f685c00c8c0 comp rx=0 tx=0).stop 2024-03-23T15:58:36.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 shutdown_connections 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f685406b810 0x7f685406dcd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68681544b0 0x7f68681f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68680caa00 0x7f68680cf930 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 --2- 172.21.15.129:0/2514685817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f68680c6000 0x7f68681593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 >> 172.21.15.129:0/2514685817 conn(0x7f6868061b50 msgr2=0x7f68680b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 shutdown_connections 2024-03-23T15:58:36.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:36.098+0000 7f686d8d4640 1 -- 172.21.15.129:0/2514685817 wait complete. 2024-03-23T15:58:36.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:36.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:37.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:37.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.821+0000 7f4e06b1e640 1 Processor -- start 2024-03-23T15:58:37.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.822+0000 7f4e06b1e640 1 -- start start 2024-03-23T15:58:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.823+0000 7f4e06b1e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4e000c6df0 0x7f4e000cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.823+0000 7f4e06b1e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e000c3770 con 0x7f4e000cc6a0 2024-03-23T15:58:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.823+0000 7f4e06b1e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e000c3a50 con 0x7f4e0013cfd0 2024-03-23T15:58:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.823+0000 7f4e06b1e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4e00157070 con 0x7f4e000c6df0 2024-03-23T15:58:37.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.824+0000 7f4e050bb640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4e000cc6a0 0x7f4e001574a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38194/0 (socket says 172.21.15.129:38194) 2024-03-23T15:58:37.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.824+0000 7f4e050bb640 1 -- 172.21.15.129:0/2973827680 learned_addr learned my addr 172.21.15.129:0/2973827680 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:37.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.824+0000 7f4dfffff640 1 --2- 172.21.15.129:0/2973827680 >> v2:172.21.15.157:3300/0 conn(0x7f4e000c6df0 0x7f4e000cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:37.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.826+0000 7f4dff7fe640 1 -- 172.21.15.129:0/2973827680 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138250481 0 0) 0x7f4e000c3770 con 0x7f4e000cc6a0 2024-03-23T15:58:37.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.826+0000 7f4dff7fe640 1 -- 172.21.15.129:0/2973827680 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df0003520 con 0x7f4e000cc6a0 2024-03-23T15:58:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.830+0000 7f4dfffff640 1 -- 172.21.15.129:0/2973827680 >> v1:172.21.15.157:6789/0 conn(0x7f4e0013cfd0 legacy=0x7f4e0006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.830+0000 7f4dfffff640 1 -- 172.21.15.129:0/2973827680 >> v1:172.21.15.129:6789/0 conn(0x7f4e000cc6a0 legacy=0x7f4e001574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.830+0000 7f4dfffff640 1 -- 172.21.15.129:0/2973827680 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e000c3d30 con 0x7f4e000c6df0 2024-03-23T15:58:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.835+0000 7f4dfffff640 1 --2- 172.21.15.129:0/2973827680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e000cc160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009790 tx=0x7f4dec02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d9a9088a6d9db3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:37.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.839+0000 7f4dff7fe640 1 -- 172.21.15.129:0/2973827680 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec030610 con 0x7f4e000c6df0 2024-03-23T15:58:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.840+0000 7f4dff7fe640 1 -- 172.21.15.129:0/2973827680 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dec030770 con 0x7f4e000c6df0 2024-03-23T15:58:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.840+0000 7f4dff7fe640 1 -- 172.21.15.129:0/2973827680 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec030ae0 con 0x7f4e000c6df0 2024-03-23T15:58:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 -- 172.21.15.129:0/2973827680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 msgr2=0x7f4e000cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/2973827680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e000cc160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009790 tx=0x7f4dec02b960 comp rx=0 tx=0).stop 2024-03-23T15:58:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 -- 172.21.15.129:0/2973827680 shutdown_connections 2024-03-23T15:58:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/2973827680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e000cc160 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 -- 172.21.15.129:0/2973827680 >> 172.21.15.129:0/2973827680 conn(0x7f4e00071dc0 msgr2=0x7f4e000721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 -- 172.21.15.129:0/2973827680 shutdown_connections 2024-03-23T15:58:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.841+0000 7f4e06b1e640 1 -- 172.21.15.129:0/2973827680 wait complete. 2024-03-23T15:58:37.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.843+0000 7f4e06b1e640 1 Processor -- start 2024-03-23T15:58:37.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.843+0000 7f4e06b1e640 1 -- start start 2024-03-23T15:58:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.844+0000 7f4e06b1e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e00153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.844+0000 7f4e06b1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e000cc6a0 0x7f4e0015aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.844+0000 7f4e06b1e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e0013cfd0 0x7f4e001f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.844+0000 7f4e06b1e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0013ba10 con 0x7f4e000cc6a0 2024-03-23T15:58:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.844+0000 7f4e06b1e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0015a3c0 con 0x7f4e000c6df0 2024-03-23T15:58:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.844+0000 7f4e06b1e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0015b2e0 con 0x7f4e0013cfd0 2024-03-23T15:58:37.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.845+0000 7f4dfffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e000cc6a0 0x7f4e0015aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:37.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.845+0000 7f4e048ba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e00153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:37.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.845+0000 7f4e050bb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e0013cfd0 0x7f4e001f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:37.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.845+0000 7f4e048ba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e00153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:50014/0 (socket says 172.21.15.129:50014) 2024-03-23T15:58:37.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.845+0000 7f4e048ba640 1 -- 172.21.15.129:0/32510087 learned_addr learned my addr 172.21.15.129:0/32510087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:37.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.849+0000 7f4e050bb640 1 -- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 msgr2=0x7f4e00153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:37.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.849+0000 7f4e050bb640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e00153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:37.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.849+0000 7f4e050bb640 1 -- 172.21.15.129:0/32510087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e000cc6a0 msgr2=0x7f4e0015aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.849+0000 7f4e050bb640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e000cc6a0 0x7f4e0015aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.849+0000 7f4e050bb640 1 -- 172.21.15.129:0/32510087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec0093a0 con 0x7f4e0013cfd0 2024-03-23T15:58:37.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.852+0000 7f4e050bb640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e0013cfd0 0x7f4e001f10e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4df800cad0 tx=0x7f4df800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:37.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.858+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df800ab70 con 0x7f4e0013cfd0 2024-03-23T15:58:37.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.858+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df800acd0 con 0x7f4e0013cfd0 2024-03-23T15:58:37.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.858+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0013bbf0 con 0x7f4e0013cfd0 2024-03-23T15:58:37.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.858+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8012600 con 0x7f4e0013cfd0 2024-03-23T15:58:37.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.858+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0013c1b0 con 0x7f4e0013cfd0 2024-03-23T15:58:37.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.860+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e00075740 con 0x7f4e0013cfd0 2024-03-23T15:58:37.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.879+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4df8007070 con 0x7f4e0013cfd0 2024-03-23T15:58:37.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.879+0000 7f4dfd7fa640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4de406b860 0x7f4de406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:37.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.879+0000 7f4e048ba640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4de406b860 0x7f4de406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:37.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:37.880+0000 7f4e048ba640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4de406b860 0x7f4de406dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4df4007fd0 tx=0x7f4df4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:38.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.116+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4df800ae40 con 0x7f4e0013cfd0 2024-03-23T15:58:38.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.135+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4df8054ff0 con 0x7f4e0013cfd0 2024-03-23T15:58:38.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.356+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e000681d0 con 0x7f4e0013cfd0 2024-03-23T15:58:38.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.372+0000 7f4dfd7fa640 1 -- 172.21.15.129:0/32510087 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4df80586b0 con 0x7f4e0013cfd0 2024-03-23T15:58:38.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:38.375 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:38.375 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:38.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4de406b860 msgr2=0x7f4de406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:38.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4de406b860 0x7f4de406dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4df4007fd0 tx=0x7f4df4009040 comp rx=0 tx=0).stop 2024-03-23T15:58:38.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e0013cfd0 msgr2=0x7f4e001f10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e0013cfd0 0x7f4e001f10e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4df800cad0 tx=0x7f4df800cf90 comp rx=0 tx=0).stop 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 shutdown_connections 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e0013cfd0 0x7f4e001f10e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e000cc6a0 0x7f4e0015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4de406b860 0x7f4de406dd20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 --2- 172.21.15.129:0/32510087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e000c6df0 0x7f4e00153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 >> 172.21.15.129:0/32510087 conn(0x7f4e00071dc0 msgr2=0x7f4e001573a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 shutdown_connections 2024-03-23T15:58:38.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:38.375+0000 7f4e06b1e640 1 -- 172.21.15.129:0/32510087 wait complete. 2024-03-23T15:58:38.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:58:38.809 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:58:38.993 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:38.993 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:39.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.093+0000 7f783f5d0640 1 Processor -- start 2024-03-23T15:58:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.094+0000 7f783f5d0640 1 -- start start 2024-03-23T15:58:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.094+0000 7f783f5d0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f783815c000 0x7f783815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.094+0000 7f783f5d0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f783815b290 con 0x7f78380ceb00 2024-03-23T15:58:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.094+0000 7f783f5d0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78380ce800 con 0x7f78380ccd60 2024-03-23T15:58:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.094+0000 7f783f5d0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7838154a70 con 0x7f783815c000 2024-03-23T15:58:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.095+0000 7f783cb6b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f78380ceb00 0x7f783815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:38196/0 (socket says 172.21.15.129:38196) 2024-03-23T15:58:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.095+0000 7f783cb6b640 1 -- 172.21.15.129:0/237199115 learned_addr learned my addr 172.21.15.129:0/237199115 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.096+0000 7f783db6d640 1 --2- 172.21.15.129:0/237199115 >> v2:172.21.15.157:3300/0 conn(0x7f783815c000 0x7f783815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:40.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.098+0000 7f782ffff640 1 -- 172.21.15.129:0/237199115 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567583153 0 0) 0x7f7828001110 con 0x7f78380ceb00 2024-03-23T15:58:40.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.098+0000 7f782ffff640 1 -- 172.21.15.129:0/237199115 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7824003520 con 0x7f78380ceb00 2024-03-23T15:58:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.102+0000 7f783db6d640 1 -- 172.21.15.129:0/237199115 >> v1:172.21.15.157:6789/0 conn(0x7f78380ccd60 legacy=0x7f783806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.102+0000 7f783db6d640 1 -- 172.21.15.129:0/237199115 >> v1:172.21.15.129:6789/0 conn(0x7f78380ceb00 legacy=0x7f783815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.102+0000 7f783db6d640 1 -- 172.21.15.129:0/237199115 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78380cd950 con 0x7f783815c000 2024-03-23T15:58:40.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.105+0000 7f783db6d640 1 --2- 172.21.15.129:0/237199115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815c000 0x7f783815ad50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7830009d90 tx=0x7f783002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=111499aafa973f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:40.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.111+0000 7f782ffff640 1 -- 172.21.15.129:0/237199115 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783002cd80 con 0x7f783815c000 2024-03-23T15:58:40.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.111+0000 7f782ffff640 1 -- 172.21.15.129:0/237199115 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7830034440 con 0x7f783815c000 2024-03-23T15:58:40.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.111+0000 7f782ffff640 1 -- 172.21.15.129:0/237199115 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300347d0 con 0x7f783815c000 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 -- 172.21.15.129:0/237199115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815c000 msgr2=0x7f783815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 --2- 172.21.15.129:0/237199115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815c000 0x7f783815ad50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7830009d90 tx=0x7f783002bfd0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 -- 172.21.15.129:0/237199115 shutdown_connections 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 --2- 172.21.15.129:0/237199115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f783815c000 0x7f783815ad50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 -- 172.21.15.129:0/237199115 >> 172.21.15.129:0/237199115 conn(0x7f78380b1840 msgr2=0x7f78380b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 -- 172.21.15.129:0/237199115 shutdown_connections 2024-03-23T15:58:40.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.112+0000 7f783f5d0640 1 -- 172.21.15.129:0/237199115 wait complete. 2024-03-23T15:58:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.113+0000 7f783f5d0640 1 Processor -- start 2024-03-23T15:58:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.113+0000 7f783f5d0640 1 -- start start 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783f5d0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78380ccd60 0x7f78381f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783f5d0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 0x7f7838158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783f5d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815c000 0x7f7838158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783f5d0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78381f45e0 con 0x7f78380ceb00 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783f5d0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f78381f4320 con 0x7f78380ccd60 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783f5d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78380cb220 con 0x7f783815c000 2024-03-23T15:58:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783cb6b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 0x7f7838158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783db6d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815c000 0x7f7838158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.114+0000 7f783cb6b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 0x7f7838158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44652/0 (socket says 172.21.15.129:44652) 2024-03-23T15:58:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.115+0000 7f783cb6b640 1 -- 172.21.15.129:0/387233845 learned_addr learned my addr 172.21.15.129:0/387233845 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.115+0000 7f783d36c640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78380ccd60 0x7f78381f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:40.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.118+0000 7f783cb6b640 1 -- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815c000 msgr2=0x7f7838158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.118+0000 7f783cb6b640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815c000 0x7f7838158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.118+0000 7f783cb6b640 1 -- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78380ccd60 msgr2=0x7f78381f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.118+0000 7f783cb6b640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78380ccd60 0x7f78381f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.118+0000 7f783cb6b640 1 -- 172.21.15.129:0/387233845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7830009a70 con 0x7f78380ceb00 2024-03-23T15:58:40.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.120+0000 7f783cb6b640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 0x7f7838158450 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f782800c700 tx=0x7f782800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:40.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.122+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828005ce0 con 0x7f78380ceb00 2024-03-23T15:58:40.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.122+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78380cb500 con 0x7f78380ceb00 2024-03-23T15:58:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.123+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7838157f30 con 0x7f78380ceb00 2024-03-23T15:58:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.123+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783804f530 con 0x7f78380ceb00 2024-03-23T15:58:40.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.334+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7828007da0 con 0x7f78380ceb00 2024-03-23T15:58:40.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.403+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828007270 con 0x7f78380ceb00 2024-03-23T15:58:40.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.403+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f78280074f0 con 0x7f78380ceb00 2024-03-23T15:58:40.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.404+0000 7f782dffb640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781c06b810 0x7f781c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:40.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.404+0000 7f783d36c640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781c06b810 0x7f781c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:40.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.405+0000 7f783d36c640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781c06b810 0x7f781c06dcd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7834000f60 tx=0x7f7834009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.417+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7828058610 con 0x7f78380ceb00 2024-03-23T15:58:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.417+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f78280583e0 con 0x7f78380ceb00 2024-03-23T15:58:40.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.663+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f783815a120 con 0x7f78380ceb00 2024-03-23T15:58:40.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.682+0000 7f782dffb640 1 -- 172.21.15.129:0/387233845 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f783815a120 con 0x7f78380ceb00 2024-03-23T15:58:40.685 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:40.685 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:40.685 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781c06b810 msgr2=0x7f781c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781c06b810 0x7f781c06dcd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7834000f60 tx=0x7f7834009040 comp rx=0 tx=0).stop 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 msgr2=0x7f7838158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 0x7f7838158450 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f782800c700 tx=0x7f782800cb00 comp rx=0 tx=0).stop 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 shutdown_connections 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f783815c000 0x7f7838158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78380ceb00 0x7f7838158450 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f781c06b810 0x7f781c06dcd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.685+0000 7f783f5d0640 1 --2- 172.21.15.129:0/387233845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78380ccd60 0x7f78381f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:40.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.686+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 >> 172.21.15.129:0/387233845 conn(0x7f78380b1840 msgr2=0x7f783815ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:40.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.686+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 shutdown_connections 2024-03-23T15:58:40.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:40.686+0000 7f783f5d0640 1 -- 172.21.15.129:0/387233845 wait complete. 2024-03-23T15:58:41.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:41.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:42.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:42.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.414+0000 7efcef163640 1 Processor -- start 2024-03-23T15:58:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.414+0000 7efcef163640 1 -- start start 2024-03-23T15:58:42.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.415+0000 7efcef163640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.415+0000 7efcef163640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efce80ceb00 0x7efce806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.415+0000 7efcef163640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efce815b290 con 0x7efce80cef40 2024-03-23T15:58:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.415+0000 7efcef163640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7efce80ce800 con 0x7efce80ceb00 2024-03-23T15:58:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.415+0000 7efcef163640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efce8154a70 con 0x7efce80ccd60 2024-03-23T15:58:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.416+0000 7efce7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efce80ceb00 0x7efce806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:42.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.416+0000 7efceceff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:42.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.416+0000 7efce7fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efce80ceb00 0x7efce806c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40710/0 (socket says 172.21.15.129:40710) 2024-03-23T15:58:42.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.416+0000 7efce7fff640 1 -- 172.21.15.129:0/3337601067 learned_addr learned my addr 172.21.15.129:0/3337601067 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:42.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.418+0000 7efce7fff640 1 -- 172.21.15.129:0/3337601067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 msgr2=0x7efce815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:42.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.418+0000 7efce7fff640 1 --2- 172.21.15.129:0/3337601067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:42.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.418+0000 7efce7fff640 1 -- 172.21.15.129:0/3337601067 >> v1:172.21.15.157:6789/0 conn(0x7efce80cef40 legacy=0x7efce815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:42.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.418+0000 7efce7fff640 1 -- 172.21.15.129:0/3337601067 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce80cd950 con 0x7efce80ceb00 2024-03-23T15:58:42.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.420+0000 7efce7fff640 1 --2- 172.21.15.129:0/3337601067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce806c270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92965833ce9a7cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.424+0000 7efce77fe640 1 -- 172.21.15.129:0/3337601067 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd402f840 con 0x7efce80ceb00 2024-03-23T15:58:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.424+0000 7efce77fe640 1 -- 172.21.15.129:0/3337601067 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcd402f9a0 con 0x7efce80ceb00 2024-03-23T15:58:42.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.424+0000 7efce77fe640 1 -- 172.21.15.129:0/3337601067 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd402fd30 con 0x7efce80ceb00 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 -- 172.21.15.129:0/3337601067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 msgr2=0x7efce806c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 --2- 172.21.15.129:0/3337601067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce806c270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd402ba20 comp rx=0 tx=0).stop 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 -- 172.21.15.129:0/3337601067 shutdown_connections 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 --2- 172.21.15.129:0/3337601067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce806c270 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 --2- 172.21.15.129:0/3337601067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 -- 172.21.15.129:0/3337601067 >> 172.21.15.129:0/3337601067 conn(0x7efce80b1840 msgr2=0x7efce80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 -- 172.21.15.129:0/3337601067 shutdown_connections 2024-03-23T15:58:42.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.425+0000 7efcef163640 1 -- 172.21.15.129:0/3337601067 wait complete. 2024-03-23T15:58:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.427+0000 7efcef163640 1 Processor -- start 2024-03-23T15:58:42.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.427+0000 7efcef163640 1 -- start start 2024-03-23T15:58:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efcef163640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce80c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efcef163640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce80c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efcef163640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce80cef40 0x7efce81f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efcef163640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efce80c6f20 con 0x7efce80ceb00 2024-03-23T15:58:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efcef163640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efce8154e50 con 0x7efce80cef40 2024-03-23T15:58:42.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efcef163640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efce80c8530 con 0x7efce80ccd60 2024-03-23T15:58:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efce7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce80c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efced700640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce80cef40 0x7efce81f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.428+0000 7efceceff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce80c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.429+0000 7efce7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce80c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40716/0 (socket says 172.21.15.129:40716) 2024-03-23T15:58:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.429+0000 7efce7fff640 1 -- 172.21.15.129:0/691931513 learned_addr learned my addr 172.21.15.129:0/691931513 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.433+0000 7efceceff640 1 -- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce80cef40 msgr2=0x7efce81f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.433+0000 7efceceff640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce80cef40 0x7efce81f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.433+0000 7efceceff640 1 -- 172.21.15.129:0/691931513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 msgr2=0x7efce80c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.433+0000 7efceceff640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce80c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:42.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.433+0000 7efceceff640 1 -- 172.21.15.129:0/691931513 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd40094d0 con 0x7efce80ccd60 2024-03-23T15:58:42.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.436+0000 7efceceff640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce80c8fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00e880 tx=0x7efcdc00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:42.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.440+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00c940 con 0x7efce80ccd60 2024-03-23T15:58:42.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.440+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcdc00caa0 con 0x7efce80ccd60 2024-03-23T15:58:42.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.440+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce80c73f0 con 0x7efce80ccd60 2024-03-23T15:58:42.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.440+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00ce50 con 0x7efce80ccd60 2024-03-23T15:58:42.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.441+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efce80c8ab0 con 0x7efce80ccd60 2024-03-23T15:58:42.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.441+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce804f530 con 0x7efce80ccd60 2024-03-23T15:58:42.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.466+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efcdc013470 con 0x7efce80ccd60 2024-03-23T15:58:42.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.466+0000 7efce57fa640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efccc06b860 0x7efccc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:42.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.466+0000 7efce7fff640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efccc06b860 0x7efccc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:42.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.467+0000 7efce7fff640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efccc06b860 0x7efccc06dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:42.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.470+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efcdc014e40 con 0x7efce80ccd60 2024-03-23T15:58:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:42.815+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcdc055560 con 0x7efce80ccd60 2024-03-23T15:58:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.037+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efce806db50 con 0x7efce80ccd60 2024-03-23T15:58:43.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.053+0000 7efce57fa640 1 -- 172.21.15.129:0/691931513 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7efcdc017020 con 0x7efce80ccd60 2024-03-23T15:58:43.056 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:43.056 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:43.056 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:43.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efccc06b860 msgr2=0x7efccc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:43.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efccc06b860 0x7efccc06dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009870 tx=0x7efcd4008030 comp rx=0 tx=0).stop 2024-03-23T15:58:43.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 msgr2=0x7efce80c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:43.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce80c8fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00e880 tx=0x7efcdc00ed40 comp rx=0 tx=0).stop 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 shutdown_connections 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efccc06b860 0x7efccc06dd20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce80cef40 0x7efce81f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce80ceb00 0x7efce80c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 --2- 172.21.15.129:0/691931513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efce80ccd60 0x7efce80c8fd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 >> 172.21.15.129:0/691931513 conn(0x7efce80b1840 msgr2=0x7efce8153e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 shutdown_connections 2024-03-23T15:58:43.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:43.056+0000 7efcef163640 1 -- 172.21.15.129:0/691931513 wait complete. 2024-03-23T15:58:43.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:43.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:44.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:44.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.776+0000 7fb144c15640 1 Processor -- start 2024-03-23T15:58:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.776+0000 7fb144c15640 1 -- start start 2024-03-23T15:58:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.777+0000 7fb144c15640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb14015d9c0 0x7fb14006c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.777+0000 7fb144c15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1400cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.777+0000 7fb144c15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1400cb810 con 0x7fb14015dda0 2024-03-23T15:58:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.777+0000 7fb144c15640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb1400cac00 con 0x7fb14015d9c0 2024-03-23T15:58:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.777+0000 7fb144c15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb1400caee0 con 0x7fb1400ca4a0 2024-03-23T15:58:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.778+0000 7fb13e59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb14015d9c0 0x7fb14006c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.778+0000 7fb13dd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1400cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.778+0000 7fb13e59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb14015d9c0 0x7fb14006c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40722/0 (socket says 172.21.15.129:40722) 2024-03-23T15:58:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.778+0000 7fb13e59c640 1 -- 172.21.15.129:0/3284415576 learned_addr learned my addr 172.21.15.129:0/3284415576 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:44.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.781+0000 7fb13dd9b640 1 -- 172.21.15.129:0/3284415576 >> v2:172.21.15.129:3300/0 conn(0x7fb14015d9c0 msgr2=0x7fb14006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:44.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.781+0000 7fb13dd9b640 1 --2- 172.21.15.129:0/3284415576 >> v2:172.21.15.129:3300/0 conn(0x7fb14015d9c0 0x7fb14006c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:44.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.781+0000 7fb13dd9b640 1 -- 172.21.15.129:0/3284415576 >> v1:172.21.15.129:6789/0 conn(0x7fb14015dda0 legacy=0x7fb14015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:44.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.781+0000 7fb13dd9b640 1 -- 172.21.15.129:0/3284415576 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1400cb1c0 con 0x7fb1400ca4a0 2024-03-23T15:58:44.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.784+0000 7fb13dd9b640 1 --2- 172.21.15.129:0/3284415576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1400cbca0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb1340097a0 tx=0x7fb13402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce335fe164ed7faf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.789+0000 7fb13d59a640 1 -- 172.21.15.129:0/3284415576 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134031830 con 0x7fb1400ca4a0 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.789+0000 7fb13d59a640 1 -- 172.21.15.129:0/3284415576 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb134031990 con 0x7fb1400ca4a0 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.789+0000 7fb13d59a640 1 -- 172.21.15.129:0/3284415576 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134031d20 con 0x7fb1400ca4a0 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 -- 172.21.15.129:0/3284415576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 msgr2=0x7fb1400cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 --2- 172.21.15.129:0/3284415576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1400cbca0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb1340097a0 tx=0x7fb13402b8d0 comp rx=0 tx=0).stop 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 -- 172.21.15.129:0/3284415576 shutdown_connections 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 --2- 172.21.15.129:0/3284415576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1400cbca0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 --2- 172.21.15.129:0/3284415576 >> v2:172.21.15.129:3300/0 conn(0x7fb14015d9c0 0x7fb14006c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 -- 172.21.15.129:0/3284415576 >> 172.21.15.129:0/3284415576 conn(0x7fb1400cfb20 msgr2=0x7fb1400cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 -- 172.21.15.129:0/3284415576 shutdown_connections 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.790+0000 7fb144c15640 1 -- 172.21.15.129:0/3284415576 wait complete. 2024-03-23T15:58:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 Processor -- start 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 -- start start 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1401e6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb14015d9c0 0x7fb1400cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb14015dda0 0x7fb1400d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400b4fe0 con 0x7fb14015dda0 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400b39d0 con 0x7fb14015d9c0 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.792+0000 7fb144c15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb14015b160 con 0x7fb1400ca4a0 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.793+0000 7fb13ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb14015dda0 0x7fb1400d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.793+0000 7fb13e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1401e6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.793+0000 7fb13dd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb14015d9c0 0x7fb1400cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.793+0000 7fb13e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1401e6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:40534/0 (socket says 172.21.15.129:40534) 2024-03-23T15:58:44.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.793+0000 7fb13e59c640 1 -- 172.21.15.129:0/3180472817 learned_addr learned my addr 172.21.15.129:0/3180472817 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.797+0000 7fb13e59c640 1 -- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb14015d9c0 msgr2=0x7fb1400cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.797+0000 7fb13e59c640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb14015d9c0 0x7fb1400cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.797+0000 7fb13e59c640 1 -- 172.21.15.129:0/3180472817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb14015dda0 msgr2=0x7fb1400d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.797+0000 7fb13e59c640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb14015dda0 0x7fb1400d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:44.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.797+0000 7fb13e59c640 1 -- 172.21.15.129:0/3180472817 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb134009400 con 0x7fb1400ca4a0 2024-03-23T15:58:44.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.800+0000 7fb13e59c640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1401e6020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb130009fd0 tx=0x7fb13000ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.804+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13000c940 con 0x7fb1400ca4a0 2024-03-23T15:58:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.804+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13000caa0 con 0x7fb1400ca4a0 2024-03-23T15:58:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.804+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1400b4b50 con 0x7fb1400ca4a0 2024-03-23T15:58:44.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.804+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13000ce30 con 0x7fb1400ca4a0 2024-03-23T15:58:44.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.804+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1400b4570 con 0x7fb1400ca4a0 2024-03-23T15:58:44.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:44.805+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1400ca920 con 0x7fb1400ca4a0 2024-03-23T15:58:45.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.065+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb130013470 con 0x7fb1400ca4a0 2024-03-23T15:58:45.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.065+0000 7fb12effd640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb12006b890 0x7fb12006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:45.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.065+0000 7fb13dd9b640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb12006b890 0x7fb12006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:45.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.066+0000 7fb13dd9b640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb12006b890 0x7fb12006dd50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb134036000 tx=0x7fb13403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:45.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.070+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb130014e40 con 0x7fb1400ca4a0 2024-03-23T15:58:45.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.081+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb130055570 con 0x7fb1400ca4a0 2024-03-23T15:58:45.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.303+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1400cf4c0 con 0x7fb1400ca4a0 2024-03-23T15:58:45.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.350+0000 7fb12effd640 1 -- 172.21.15.129:0/3180472817 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb130017090 con 0x7fb1400ca4a0 2024-03-23T15:58:45.353 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:45.353 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:45.353 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb12006b890 msgr2=0x7fb12006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb12006b890 0x7fb12006dd50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb134036000 tx=0x7fb13403c040 comp rx=0 tx=0).stop 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 msgr2=0x7fb1401e6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1401e6020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb130009fd0 tx=0x7fb13000ed40 comp rx=0 tx=0).stop 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 shutdown_connections 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb14015dda0 0x7fb1400d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb12006b890 0x7fb12006dd50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb14015d9c0 0x7fb1400cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 --2- 172.21.15.129:0/3180472817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1400ca4a0 0x7fb1401e6020 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 >> 172.21.15.129:0/3180472817 conn(0x7fb1400cfb20 msgr2=0x7fb1400c73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 shutdown_connections 2024-03-23T15:58:45.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:45.352+0000 7fb144c15640 1 -- 172.21.15.129:0/3180472817 wait complete. 2024-03-23T15:58:45.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:45.970 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:46.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:47.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.071+0000 7f5094e9a640 1 Processor -- start 2024-03-23T15:58:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.072+0000 7f5094e9a640 1 -- start start 2024-03-23T15:58:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.072+0000 7f5094e9a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f509006c410 0x7f509015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.072+0000 7f5094e9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509015de40 0x7f509015e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.072+0000 7f5094e9a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509006c830 con 0x7f509006c030 2024-03-23T15:58:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.072+0000 7f5094e9a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f50900cc370 con 0x7f509006c410 2024-03-23T15:58:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.072+0000 7f5094e9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f509006cae0 con 0x7f509015de40 2024-03-23T15:58:47.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.073+0000 7f508e59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f509006c030 0x7f50900cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:35158/0 (socket says 172.21.15.129:35158) 2024-03-23T15:58:47.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.073+0000 7f508e59c640 1 -- 172.21.15.129:0/965454581 learned_addr learned my addr 172.21.15.129:0/965454581 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:47.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.073+0000 7f508ed9d640 1 --2- 172.21.15.129:0/965454581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509015de40 0x7f509015e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.075+0000 7f508d59a640 1 -- 172.21.15.129:0/965454581 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998816827 0 0) 0x7f509006c830 con 0x7f509006c030 2024-03-23T15:58:47.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.075+0000 7f508d59a640 1 -- 172.21.15.129:0/965454581 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5078003520 con 0x7f509006c030 2024-03-23T15:58:47.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.076+0000 7f508dd9b640 1 --2- 172.21.15.129:0/965454581 >> v2:172.21.15.157:3300/0 conn(0x7f509006c410 0x7f509015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.078+0000 7f508ed9d640 1 -- 172.21.15.129:0/965454581 >> v2:172.21.15.157:3300/0 conn(0x7f509006c410 msgr2=0x7f509015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.078+0000 7f508ed9d640 1 --2- 172.21.15.129:0/965454581 >> v2:172.21.15.157:3300/0 conn(0x7f509006c410 0x7f509015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.078+0000 7f508ed9d640 1 -- 172.21.15.129:0/965454581 >> v1:172.21.15.129:6789/0 conn(0x7f509006c030 legacy=0x7f50900cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.078+0000 7f508ed9d640 1 -- 172.21.15.129:0/965454581 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50900cabc0 con 0x7f509015de40 2024-03-23T15:58:47.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.082+0000 7f508ed9d640 1 --2- 172.21.15.129:0/965454581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509015de40 0x7f509015e2b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5088009d90 tx=0x7f508802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58254a510fd04f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:47.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.087+0000 7f508d59a640 1 -- 172.21.15.129:0/965454581 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508802cc50 con 0x7f509015de40 2024-03-23T15:58:47.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.087+0000 7f508d59a640 1 -- 172.21.15.129:0/965454581 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f508802cdb0 con 0x7f509015de40 2024-03-23T15:58:47.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.087+0000 7f508d59a640 1 -- 172.21.15.129:0/965454581 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088034660 con 0x7f509015de40 2024-03-23T15:58:47.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 -- 172.21.15.129:0/965454581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509015de40 msgr2=0x7f509015e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:47.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 --2- 172.21.15.129:0/965454581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509015de40 0x7f509015e2b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5088009d90 tx=0x7f508802bea0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 -- 172.21.15.129:0/965454581 shutdown_connections 2024-03-23T15:58:47.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 --2- 172.21.15.129:0/965454581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509015de40 0x7f509015e2b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 --2- 172.21.15.129:0/965454581 >> v2:172.21.15.157:3300/0 conn(0x7f509006c410 0x7f509015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 -- 172.21.15.129:0/965454581 >> 172.21.15.129:0/965454581 conn(0x7f50900cfb20 msgr2=0x7f50900cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:47.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 -- 172.21.15.129:0/965454581 shutdown_connections 2024-03-23T15:58:47.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.088+0000 7f5094e9a640 1 -- 172.21.15.129:0/965454581 wait complete. 2024-03-23T15:58:47.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.090+0000 7f5094e9a640 1 Processor -- start 2024-03-23T15:58:47.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.090+0000 7f5094e9a640 1 -- start start 2024-03-23T15:58:47.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.091+0000 7f5094e9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f509006c030 0x7f5090157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:47.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.091+0000 7f5094e9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509006c410 0x7f50901f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.091+0000 7f5094e9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f509015de40 0x7f50900b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.091+0000 7f5094e9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5090157b40 con 0x7f509015de40 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.091+0000 7f5094e9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f50901f4180 con 0x7f509006c030 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.091+0000 7f5094e9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5090154e20 con 0x7f509006c410 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.092+0000 7f508ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f509015de40 0x7f50900b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.092+0000 7f508e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f509006c030 0x7f5090157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.092+0000 7f508dd9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509006c410 0x7f50901f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.092+0000 7f508ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f509015de40 0x7f50900b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40760/0 (socket says 172.21.15.129:40760) 2024-03-23T15:58:47.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.092+0000 7f508ed9d640 1 -- 172.21.15.129:0/2496882018 learned_addr learned my addr 172.21.15.129:0/2496882018 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:47.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.096+0000 7f508e59c640 1 -- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509006c410 msgr2=0x7f50901f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:47.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.096+0000 7f508e59c640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509006c410 0x7f50901f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.096+0000 7f508e59c640 1 -- 172.21.15.129:0/2496882018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f509015de40 msgr2=0x7f50900b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:47.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.096+0000 7f508e59c640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f509015de40 0x7f50900b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.096+0000 7f508e59c640 1 -- 172.21.15.129:0/2496882018 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5088009a70 con 0x7f509006c030 2024-03-23T15:58:47.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.099+0000 7f508e59c640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f509006c030 0x7f5090157600 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f508000c400 tx=0x7f508000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:47.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.105+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080005ce0 con 0x7f509006c030 2024-03-23T15:58:47.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.105+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5080005e40 con 0x7f509006c030 2024-03-23T15:58:47.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.105+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5090155100 con 0x7f509006c030 2024-03-23T15:58:47.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.105+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50800072f0 con 0x7f509006c030 2024-03-23T15:58:47.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.105+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901f01d0 con 0x7f509006c030 2024-03-23T15:58:47.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.106+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509006b8a0 con 0x7f509006c030 2024-03-23T15:58:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.138+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5080007450 con 0x7f509006c030 2024-03-23T15:58:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.138+0000 7f507effd640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f507006b860 0x7f507006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:47.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.138+0000 7f508dd9b640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f507006b860 0x7f507006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:47.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.139+0000 7f508dd9b640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f507006b860 0x7f507006dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f50900b5740 tx=0x7f5084009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.380+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5080007e40 con 0x7f509006c030 2024-03-23T15:58:47.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.400+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f50800545b0 con 0x7f509006c030 2024-03-23T15:58:47.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.622+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5090075740 con 0x7f509006c030 2024-03-23T15:58:47.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.670+0000 7f507effd640 1 -- 172.21.15.129:0/2496882018 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5080017070 con 0x7f509006c030 2024-03-23T15:58:47.673 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:47.674 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:47.674 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:47.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f507006b860 msgr2=0x7f507006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:47.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f507006b860 0x7f507006dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f50900b5740 tx=0x7f5084009190 comp rx=0 tx=0).stop 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f509006c030 msgr2=0x7f5090157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f509006c030 0x7f5090157600 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f508000c400 tx=0x7f508000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 shutdown_connections 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f509015de40 0x7f50900b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f509006c410 0x7f50901f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f507006b860 0x7f507006dd20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 --2- 172.21.15.129:0/2496882018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f509006c030 0x7f5090157600 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 >> 172.21.15.129:0/2496882018 conn(0x7f50900cfb20 msgr2=0x7f50900c73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:47.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 shutdown_connections 2024-03-23T15:58:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:47.672+0000 7f5094e9a640 1 -- 172.21.15.129:0/2496882018 wait complete. 2024-03-23T15:58:48.291 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:48.291 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:49.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:49.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.393+0000 7fe1817c9640 1 Processor -- start 2024-03-23T15:58:49.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.393+0000 7fe1817c9640 1 -- start start 2024-03-23T15:58:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.394+0000 7fe1817c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe17c1544b0 0x7fe17c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.394+0000 7fe1817c9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe17c0caa00 0x7fe17c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.394+0000 7fe1817c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c0cbcb0 con 0x7fe17c06c0b0 2024-03-23T15:58:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.394+0000 7fe1817c9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe17c0c5be0 con 0x7fe17c0caa00 2024-03-23T15:58:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.394+0000 7fe1817c9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe17c15b280 con 0x7fe17c1544b0 2024-03-23T15:58:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.394+0000 7fe17a7fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe17c0caa00 0x7fe17c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.395+0000 7fe17affd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe17c1544b0 0x7fe17c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.395+0000 7fe17b7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe17c06c0b0 0x7fe17c0c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:38406/0 (socket says 172.21.15.129:38406) 2024-03-23T15:58:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.395+0000 7fe17b7fe640 1 -- 172.21.15.129:0/3761223575 learned_addr learned my addr 172.21.15.129:0/3761223575 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.397+0000 7fe17a7fc640 1 -- 172.21.15.129:0/3761223575 >> v2:172.21.15.157:3300/0 conn(0x7fe17c1544b0 msgr2=0x7fe17c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.397+0000 7fe17a7fc640 1 --2- 172.21.15.129:0/3761223575 >> v2:172.21.15.157:3300/0 conn(0x7fe17c1544b0 0x7fe17c15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.397+0000 7fe17a7fc640 1 -- 172.21.15.129:0/3761223575 >> v1:172.21.15.157:6789/0 conn(0x7fe17c06c0b0 legacy=0x7fe17c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.397+0000 7fe17a7fc640 1 -- 172.21.15.129:0/3761223575 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c15a9e0 con 0x7fe17c0caa00 2024-03-23T15:58:49.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.399+0000 7fe17a7fc640 1 --2- 172.21.15.129:0/3761223575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c0caa00 0x7fe17c06bb70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe17c06cd70 tx=0x7fe17002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28a7e8dface0360 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.401+0000 7fe179ffb640 1 -- 172.21.15.129:0/3761223575 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170030610 con 0x7fe17c0caa00 2024-03-23T15:58:49.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.402+0000 7fe179ffb640 1 -- 172.21.15.129:0/3761223575 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe170030770 con 0x7fe17c0caa00 2024-03-23T15:58:49.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.402+0000 7fe179ffb640 1 -- 172.21.15.129:0/3761223575 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170030b00 con 0x7fe17c0caa00 2024-03-23T15:58:49.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 -- 172.21.15.129:0/3761223575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c0caa00 msgr2=0x7fe17c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 --2- 172.21.15.129:0/3761223575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c0caa00 0x7fe17c06bb70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe17c06cd70 tx=0x7fe17002b960 comp rx=0 tx=0).stop 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 -- 172.21.15.129:0/3761223575 shutdown_connections 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 --2- 172.21.15.129:0/3761223575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c0caa00 0x7fe17c06bb70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 --2- 172.21.15.129:0/3761223575 >> v2:172.21.15.157:3300/0 conn(0x7fe17c1544b0 0x7fe17c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 -- 172.21.15.129:0/3761223575 >> 172.21.15.129:0/3761223575 conn(0x7fe17c061b50 msgr2=0x7fe17c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 -- 172.21.15.129:0/3761223575 shutdown_connections 2024-03-23T15:58:49.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.403+0000 7fe1817c9640 1 -- 172.21.15.129:0/3761223575 wait complete. 2024-03-23T15:58:49.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 Processor -- start 2024-03-23T15:58:49.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 -- start start 2024-03-23T15:58:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe17c06c0b0 0x7fe17c1486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe17c0caa00 0x7fe17c1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 0x7fe17c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c147b80 con 0x7fe17c1544b0 2024-03-23T15:58:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c146cb0 con 0x7fe17c06c0b0 2024-03-23T15:58:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.405+0000 7fe1817c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c146f70 con 0x7fe17c0caa00 2024-03-23T15:58:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.406+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 0x7fe17c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.406+0000 7fe17affd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe17c06c0b0 0x7fe17c1486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.406+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 0x7fe17c156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40780/0 (socket says 172.21.15.129:40780) 2024-03-23T15:58:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.406+0000 7fe17b7fe640 1 -- 172.21.15.129:0/4227565863 learned_addr learned my addr 172.21.15.129:0/4227565863 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:49.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.406+0000 7fe17a7fc640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe17c0caa00 0x7fe17c1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:49.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.409+0000 7fe17b7fe640 1 -- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe17c0caa00 msgr2=0x7fe17c1594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:49.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.409+0000 7fe17b7fe640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe17c0caa00 0x7fe17c1594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.409+0000 7fe17b7fe640 1 -- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe17c06c0b0 msgr2=0x7fe17c1486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:49.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.409+0000 7fe17b7fe640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe17c06c0b0 0x7fe17c1486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.409+0000 7fe17b7fe640 1 -- 172.21.15.129:0/4227565863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe170009400 con 0x7fe17c1544b0 2024-03-23T15:58:49.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.411+0000 7fe17b7fe640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 0x7fe17c156a80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe17400cbc0 tx=0x7fe17400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.415+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17400ad40 con 0x7fe17c1544b0 2024-03-23T15:58:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.415+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe174012400 con 0x7fe17c1544b0 2024-03-23T15:58:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.415+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c147250 con 0x7fe17c1544b0 2024-03-23T15:58:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.415+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17400aea0 con 0x7fe17c1544b0 2024-03-23T15:58:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.415+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c147810 con 0x7fe17c1544b0 2024-03-23T15:58:49.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.416+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c1567a0 con 0x7fe17c1544b0 2024-03-23T15:58:49.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.671+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe174007070 con 0x7fe17c1544b0 2024-03-23T15:58:49.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.672+0000 7fe1637fe640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe16806b810 0x7fe16806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:49.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.672+0000 7fe17affd640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe16806b810 0x7fe16806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:49.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.673+0000 7fe17affd640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe16806b810 0x7fe16806dcd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009aa0 tx=0x7fe16c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.685+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe174058c50 con 0x7fe17c1544b0 2024-03-23T15:58:49.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.685+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe174054ef0 con 0x7fe17c1544b0 2024-03-23T15:58:49.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.905+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe17c1567a0 con 0x7fe17c1544b0 2024-03-23T15:58:49.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.926+0000 7fe1637fe640 1 -- 172.21.15.129:0/4227565863 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe17c1567a0 con 0x7fe17c1544b0 2024-03-23T15:58:49.929 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:49.929 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:49.929 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:49.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe16806b810 msgr2=0x7fe16806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:49.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe16806b810 0x7fe16806dcd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009aa0 tx=0x7fe16c009190 comp rx=0 tx=0).stop 2024-03-23T15:58:49.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 msgr2=0x7fe17c156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:49.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 0x7fe17c156a80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe17400cbc0 tx=0x7fe17400a430 comp rx=0 tx=0).stop 2024-03-23T15:58:49.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 shutdown_connections 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c1544b0 0x7fe17c156a80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe17c0caa00 0x7fe17c1594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe16806b810 0x7fe16806dcd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 --2- 172.21.15.129:0/4227565863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe17c06c0b0 0x7fe17c1486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 >> 172.21.15.129:0/4227565863 conn(0x7fe17c061b50 msgr2=0x7fe17c0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 shutdown_connections 2024-03-23T15:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:49.929+0000 7fe1817c9640 1 -- 172.21.15.129:0/4227565863 wait complete. 2024-03-23T15:58:50.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:50.547 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:51.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:51.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.648+0000 7f3c83f2e640 1 Processor -- start 2024-03-23T15:58:51.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.648+0000 7f3c83f2e640 1 -- start start 2024-03-23T15:58:51.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.649+0000 7f3c83f2e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ccd60 0x7f3c7c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:51.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.649+0000 7f3c83f2e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3c7c0ceb00 0x7f3c7c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:51.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.649+0000 7f3c83f2e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c7c15b290 con 0x7f3c7c0cef40 2024-03-23T15:58:51.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.649+0000 7f3c83f2e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3c7c0ce800 con 0x7f3c7c0ceb00 2024-03-23T15:58:51.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.649+0000 7f3c83f2e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3c7c154a70 con 0x7f3c7c0ccd60 2024-03-23T15:58:51.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.650+0000 7f3c81cca640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ccd60 0x7f3c7c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:51.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.650+0000 7f3c814c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3c7c0ceb00 0x7f3c7c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:51.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.650+0000 7f3c824cb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3c7c0cef40 0x7f3c7c15b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:40254/0 (socket says 172.21.15.129:40254) 2024-03-23T15:58:51.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.650+0000 7f3c824cb640 1 -- 172.21.15.129:0/3807575719 learned_addr learned my addr 172.21.15.129:0/3807575719 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:51.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.654+0000 7f3c81cca640 1 -- 172.21.15.129:0/3807575719 >> v2:172.21.15.157:3300/0 conn(0x7f3c7c0ceb00 msgr2=0x7f3c7c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:51.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.654+0000 7f3c81cca640 1 --2- 172.21.15.129:0/3807575719 >> v2:172.21.15.157:3300/0 conn(0x7f3c7c0ceb00 0x7f3c7c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:51.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.654+0000 7f3c81cca640 1 -- 172.21.15.129:0/3807575719 >> v1:172.21.15.157:6789/0 conn(0x7f3c7c0cef40 legacy=0x7f3c7c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:51.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.654+0000 7f3c81cca640 1 -- 172.21.15.129:0/3807575719 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7c0cd950 con 0x7f3c7c0ccd60 2024-03-23T15:58:51.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.656+0000 7f3c81cca640 1 --2- 172.21.15.129:0/3807575719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ccd60 0x7f3c7c15ad50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3c780097a0 tx=0x7f3c7802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff700da0ad612bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c80cc8640 1 -- 172.21.15.129:0/3807575719 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78030610 con 0x7f3c7c0ccd60 2024-03-23T15:58:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c80cc8640 1 -- 172.21.15.129:0/3807575719 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c78030770 con 0x7f3c7c0ccd60 2024-03-23T15:58:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c80cc8640 1 -- 172.21.15.129:0/3807575719 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78030b00 con 0x7f3c7c0ccd60 2024-03-23T15:58:51.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3807575719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ccd60 msgr2=0x7f3c7c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:51.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3807575719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ccd60 0x7f3c7c15ad50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3c780097a0 tx=0x7f3c7802b960 comp rx=0 tx=0).stop 2024-03-23T15:58:51.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3807575719 shutdown_connections 2024-03-23T15:58:51.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3807575719 >> v2:172.21.15.157:3300/0 conn(0x7f3c7c0ceb00 0x7f3c7c06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:51.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3807575719 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ccd60 0x7f3c7c15ad50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:51.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.661+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3807575719 >> 172.21.15.129:0/3807575719 conn(0x7f3c7c0b1840 msgr2=0x7f3c7c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.662+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3807575719 shutdown_connections 2024-03-23T15:58:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.662+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3807575719 wait complete. 2024-03-23T15:58:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 Processor -- start 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 -- start start 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c7c0ccd60 0x7f3c7c1577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ceb00 0x7f3c7c15cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 0x7f3c7c0c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1f11d0 con 0x7f3c7c0cef40 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1f03b0 con 0x7f3c7c0ccd60 2024-03-23T15:58:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.663+0000 7f3c83f2e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3c7c1f0650 con 0x7f3c7c0ceb00 2024-03-23T15:58:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.664+0000 7f3c814c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ceb00 0x7f3c7c15cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.664+0000 7f3c824cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 0x7f3c7c0c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.664+0000 7f3c81cca640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c7c0ccd60 0x7f3c7c1577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.664+0000 7f3c824cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 0x7f3c7c0c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46420/0 (socket says 172.21.15.129:46420) 2024-03-23T15:58:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.664+0000 7f3c824cb640 1 -- 172.21.15.129:0/3538501255 learned_addr learned my addr 172.21.15.129:0/3538501255 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.668+0000 7f3c824cb640 1 -- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ceb00 msgr2=0x7f3c7c15cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.668+0000 7f3c824cb640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ceb00 0x7f3c7c15cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.668+0000 7f3c824cb640 1 -- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c7c0ccd60 msgr2=0x7f3c7c1577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.668+0000 7f3c824cb640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c7c0ccd60 0x7f3c7c1577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.668+0000 7f3c824cb640 1 -- 172.21.15.129:0/3538501255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c78009400 con 0x7f3c7c0cef40 2024-03-23T15:58:51.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.670+0000 7f3c824cb640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 0x7f3c7c0c5e80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400ca80 tx=0x7f3c7400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:51.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.672+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7400ac30 con 0x7f3c7c0cef40 2024-03-23T15:58:51.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.672+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7c1f0930 con 0x7f3c7c0cef40 2024-03-23T15:58:51.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.672+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7c1f0ef0 con 0x7f3c7c0cef40 2024-03-23T15:58:51.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.674+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7c04f530 con 0x7f3c7c0cef40 2024-03-23T15:58:51.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.674+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c74001b40 con 0x7f3c7c0cef40 2024-03-23T15:58:51.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.674+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7401c440 con 0x7f3c7c0cef40 2024-03-23T15:58:51.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.940+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c74012400 con 0x7f3c7c0cef40 2024-03-23T15:58:51.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.940+0000 7f3c727fc640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c6006b840 0x7f3c6006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:51.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.940+0000 7f3c81cca640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c6006b840 0x7f3c6006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:51.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.941+0000 7f3c81cca640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c6006b840 0x7f3c6006dd00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c78036000 tx=0x7f3c78008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:51.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.963+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3c74058ee0 con 0x7f3c7c0cef40 2024-03-23T15:58:51.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:51.963+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c74055180 con 0x7f3c7c0cef40 2024-03-23T15:58:52.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.227+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c7c0686f0 con 0x7f3c7c0cef40 2024-03-23T15:58:52.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.240+0000 7f3c727fc640 1 -- 172.21.15.129:0/3538501255 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3c74058840 con 0x7f3c7c0cef40 2024-03-23T15:58:52.242 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:52.242 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:52.242 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.242+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c6006b840 msgr2=0x7f3c6006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.242+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c6006b840 0x7f3c6006dd00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c78036000 tx=0x7f3c78008030 comp rx=0 tx=0).stop 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.242+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 msgr2=0x7f3c7c0c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.242+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 0x7f3c7c0c5e80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400ca80 tx=0x7f3c7400cf40 comp rx=0 tx=0).stop 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.242+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 shutdown_connections 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.242+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c0cef40 0x7f3c7c0c5e80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.243+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c7c0ceb00 0x7f3c7c15cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.243+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c6006b840 0x7f3c6006dd00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:52.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.243+0000 7f3c83f2e640 1 --2- 172.21.15.129:0/3538501255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c7c0ccd60 0x7f3c7c1577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:52.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.243+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 >> 172.21.15.129:0/3538501255 conn(0x7f3c7c0b1840 msgr2=0x7f3c7c0cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:52.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.243+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 shutdown_connections 2024-03-23T15:58:52.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:52.243+0000 7f3c83f2e640 1 -- 172.21.15.129:0/3538501255 wait complete. 2024-03-23T15:58:52.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:52.860 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:53.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.962+0000 7f644d7af640 1 Processor -- start 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.962+0000 7f644d7af640 1 -- start start 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.963+0000 7f644d7af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.963+0000 7f644d7af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f644815dda0 0x7f644815e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.963+0000 7f644d7af640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64480cb810 con 0x7f64480ca4a0 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.963+0000 7f644d7af640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f644815e770 con 0x7f644815dda0 2024-03-23T15:58:53.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.963+0000 7f644d7af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f64480cc3b0 con 0x7f644815d9c0 2024-03-23T15:58:53.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.963+0000 7f6446ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:53.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.964+0000 7f64477fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f644815dda0 0x7f644815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:53.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.964+0000 7f643ffff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f64480ca4a0 0x7f64480cbca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:51506/0 (socket says 172.21.15.129:51506) 2024-03-23T15:58:53.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.964+0000 7f643ffff640 1 -- 172.21.15.129:0/3740255040 learned_addr learned my addr 172.21.15.129:0/3740255040 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:53.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.966+0000 7f6446ffd640 1 -- 172.21.15.129:0/3740255040 >> v2:172.21.15.129:3300/0 conn(0x7f644815dda0 msgr2=0x7f644815e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:53.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.966+0000 7f6446ffd640 1 --2- 172.21.15.129:0/3740255040 >> v2:172.21.15.129:3300/0 conn(0x7f644815dda0 0x7f644815e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:53.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.966+0000 7f6446ffd640 1 -- 172.21.15.129:0/3740255040 >> v1:172.21.15.129:6789/0 conn(0x7f64480ca4a0 legacy=0x7f64480cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:53.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.966+0000 7f6446ffd640 1 -- 172.21.15.129:0/3740255040 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64480cac00 con 0x7f644815d9c0 2024-03-23T15:58:53.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.968+0000 7f6446ffd640 1 --2- 172.21.15.129:0/3740255040 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644806c5a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f64380097a0 tx=0x7f643802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca4e3a7002879461 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:53.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.973+0000 7f64467fc640 1 -- 172.21.15.129:0/3740255040 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438031830 con 0x7f644815d9c0 2024-03-23T15:58:53.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.973+0000 7f64467fc640 1 -- 172.21.15.129:0/3740255040 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6438031990 con 0x7f644815d9c0 2024-03-23T15:58:53.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.973+0000 7f64467fc640 1 -- 172.21.15.129:0/3740255040 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438031d20 con 0x7f644815d9c0 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 -- 172.21.15.129:0/3740255040 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 msgr2=0x7f644806c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 --2- 172.21.15.129:0/3740255040 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644806c5a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f64380097a0 tx=0x7f643802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 -- 172.21.15.129:0/3740255040 shutdown_connections 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 --2- 172.21.15.129:0/3740255040 >> v2:172.21.15.129:3300/0 conn(0x7f644815dda0 0x7f644815e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 --2- 172.21.15.129:0/3740255040 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644806c5a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 -- 172.21.15.129:0/3740255040 >> 172.21.15.129:0/3740255040 conn(0x7f64480cfb20 msgr2=0x7f64480cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 -- 172.21.15.129:0/3740255040 shutdown_connections 2024-03-23T15:58:53.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.974+0000 7f644d7af640 1 -- 172.21.15.129:0/3740255040 wait complete. 2024-03-23T15:58:53.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.976+0000 7f644d7af640 1 Processor -- start 2024-03-23T15:58:53.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.976+0000 7f644d7af640 1 -- start start 2024-03-23T15:58:53.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f644d7af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64480ca4a0 0x7f644814ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:53.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f644d7af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644814d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:53.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f644d7af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644815dda0 0x7f64481f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:53.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f644d7af640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64481e51f0 con 0x7f644815dda0 2024-03-23T15:58:53.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f644d7af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f64481e5d10 con 0x7f64480ca4a0 2024-03-23T15:58:53.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f644d7af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f644814fa20 con 0x7f644815d9c0 2024-03-23T15:58:53.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.977+0000 7f64477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644815dda0 0x7f64481f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:53.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.978+0000 7f643ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644814d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:53.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.978+0000 7f6446ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64480ca4a0 0x7f644814ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:53.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.978+0000 7f64477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644815dda0 0x7f64481f0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46446/0 (socket says 172.21.15.129:46446) 2024-03-23T15:58:53.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.978+0000 7f64477fe640 1 -- 172.21.15.129:0/2698779346 learned_addr learned my addr 172.21.15.129:0/2698779346 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:53.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.978+0000 7f643ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644814d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49044/0 (socket says 172.21.15.129:49044) 2024-03-23T15:58:53.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.982+0000 7f643ffff640 1 -- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64480ca4a0 msgr2=0x7f644814ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.982+0000 7f643ffff640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64480ca4a0 0x7f644814ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.982+0000 7f643ffff640 1 -- 172.21.15.129:0/2698779346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644815dda0 msgr2=0x7f64481f0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.982+0000 7f643ffff640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644815dda0 0x7f64481f0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.982+0000 7f643ffff640 1 -- 172.21.15.129:0/2698779346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6438009400 con 0x7f644815d9c0 2024-03-23T15:58:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.986+0000 7f643ffff640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644814d8c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6430000f60 tx=0x7f643000c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.991+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6430005ce0 con 0x7f644815d9c0 2024-03-23T15:58:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.991+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6430005e40 con 0x7f644815d9c0 2024-03-23T15:58:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.991+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6430007310 con 0x7f644815d9c0 2024-03-23T15:58:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.991+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644814f740 con 0x7f644815d9c0 2024-03-23T15:58:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.991+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64481e56c0 con 0x7f644815d9c0 2024-03-23T15:58:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:53.992+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64480d09c0 con 0x7f644815d9c0 2024-03-23T15:58:54.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.300+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6430007470 con 0x7f644815d9c0 2024-03-23T15:58:54.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.301+0000 7f643f7fe640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f642806b890 0x7f642806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:54.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.301+0000 7f6446ffd640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f642806b890 0x7f642806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:54.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.302+0000 7f6446ffd640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f642806b890 0x7f642806dd50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6438036000 tx=0x7f643803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:54.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.305+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6430001000 con 0x7f644815d9c0 2024-03-23T15:58:54.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.322+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6430054510 con 0x7f644815d9c0 2024-03-23T15:58:54.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.538+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64480b5aa0 con 0x7f644815d9c0 2024-03-23T15:58:54.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.560+0000 7f643f7fe640 1 -- 172.21.15.129:0/2698779346 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6430017070 con 0x7f644815d9c0 2024-03-23T15:58:54.562 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:54.563 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:54.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f642806b890 msgr2=0x7f642806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:54.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f642806b890 0x7f642806dd50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6438036000 tx=0x7f643803c040 comp rx=0 tx=0).stop 2024-03-23T15:58:54.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 msgr2=0x7f644814d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644814d8c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6430000f60 tx=0x7f643000c760 comp rx=0 tx=0).stop 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 shutdown_connections 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f644815dda0 0x7f64481f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f644815d9c0 0x7f644814d8c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f642806b890 0x7f642806dd50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 --2- 172.21.15.129:0/2698779346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f64480ca4a0 0x7f644814ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 >> 172.21.15.129:0/2698779346 conn(0x7f64480cfb20 msgr2=0x7f64480c6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 shutdown_connections 2024-03-23T15:58:54.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:54.562+0000 7f644d7af640 1 -- 172.21.15.129:0/2698779346 wait complete. 2024-03-23T15:58:55.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:55.180 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:56.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.281+0000 7f30c3aeb640 1 Processor -- start 2024-03-23T15:58:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.281+0000 7f30c3aeb640 1 -- start start 2024-03-23T15:58:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c3aeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c3aeb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f30bc0c8f70 0x7f30bc15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c3aeb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30bc1591e0 con 0x7f30bc071530 2024-03-23T15:58:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c3aeb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f30bc141a10 con 0x7f30bc0c8f70 2024-03-23T15:58:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c3aeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f30bc15bba0 con 0x7f30bc141630 2024-03-23T15:58:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c2088640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f30bc071530 0x7f30bc0c4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:40264/0 (socket says 172.21.15.129:40264) 2024-03-23T15:58:56.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c2088640 1 -- 172.21.15.129:0/4122785373 learned_addr learned my addr 172.21.15.129:0/4122785373 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:58:56.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.282+0000 7f30c1086640 1 --2- 172.21.15.129:0/4122785373 >> v2:172.21.15.129:3300/0 conn(0x7f30bc0c8f70 0x7f30bc15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:56.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.283+0000 7f30c1887640 1 --2- 172.21.15.129:0/4122785373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.285+0000 7f30c0885640 1 -- 172.21.15.129:0/4122785373 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679510149 0 0) 0x7f30b4001490 con 0x7f30bc071530 2024-03-23T15:58:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.285+0000 7f30c0885640 1 -- 172.21.15.129:0/4122785373 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30a8003520 con 0x7f30bc071530 2024-03-23T15:58:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.286+0000 7f30c1086640 1 -- 172.21.15.129:0/4122785373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 msgr2=0x7f30bc158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.286+0000 7f30c1086640 1 --2- 172.21.15.129:0/4122785373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.286+0000 7f30c1086640 1 -- 172.21.15.129:0/4122785373 >> v1:172.21.15.157:6789/0 conn(0x7f30bc071530 legacy=0x7f30bc0c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.286+0000 7f30c1086640 1 -- 172.21.15.129:0/4122785373 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc071b00 con 0x7f30bc0c8f70 2024-03-23T15:58:56.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.288+0000 7f30c1086640 1 --2- 172.21.15.129:0/4122785373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc15c030 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0097a0 tx=0x7f30ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9462682f3fef61a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.292+0000 7f30c0885640 1 -- 172.21.15.129:0/4122785373 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac030610 con 0x7f30bc0c8f70 2024-03-23T15:58:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.292+0000 7f30c0885640 1 -- 172.21.15.129:0/4122785373 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ac030770 con 0x7f30bc0c8f70 2024-03-23T15:58:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.292+0000 7f30c0885640 1 -- 172.21.15.129:0/4122785373 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac030b00 con 0x7f30bc0c8f70 2024-03-23T15:58:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 -- 172.21.15.129:0/4122785373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 msgr2=0x7f30bc15c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/4122785373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc15c030 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0097a0 tx=0x7f30ac02b960 comp rx=0 tx=0).stop 2024-03-23T15:58:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 -- 172.21.15.129:0/4122785373 shutdown_connections 2024-03-23T15:58:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/4122785373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc15c030 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/4122785373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 -- 172.21.15.129:0/4122785373 >> 172.21.15.129:0/4122785373 conn(0x7f30bc0c8370 msgr2=0x7f30bc0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 -- 172.21.15.129:0/4122785373 shutdown_connections 2024-03-23T15:58:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.293+0000 7f30c3aeb640 1 -- 172.21.15.129:0/4122785373 wait complete. 2024-03-23T15:58:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.295+0000 7f30c3aeb640 1 Processor -- start 2024-03-23T15:58:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.295+0000 7f30c3aeb640 1 -- start start 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.296+0000 7f30c3aeb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30bc071530 0x7f30bc1f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.296+0000 7f30c3aeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc1ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.296+0000 7f30c3aeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc1400b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.296+0000 7f30c3aeb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc157160 con 0x7f30bc0c8f70 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.296+0000 7f30c3aeb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc153510 con 0x7f30bc071530 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.296+0000 7f30c3aeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f30bc153740 con 0x7f30bc141630 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.297+0000 7f30c1086640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc1ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.297+0000 7f30c1887640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30bc071530 0x7f30bc1f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.297+0000 7f30c1086640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc1ee200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46462/0 (socket says 172.21.15.129:46462) 2024-03-23T15:58:56.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.297+0000 7f30c1086640 1 -- 172.21.15.129:0/2186584869 learned_addr learned my addr 172.21.15.129:0/2186584869 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:56.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.297+0000 7f30c2088640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc1400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:56.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.300+0000 7f30c1086640 1 -- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 msgr2=0x7f30bc1400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:56.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.300+0000 7f30c1086640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc1400b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.300+0000 7f30c1086640 1 -- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30bc071530 msgr2=0x7f30bc1f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:56.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.300+0000 7f30c1086640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30bc071530 0x7f30bc1f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.300+0000 7f30c1086640 1 -- 172.21.15.129:0/2186584869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac009400 con 0x7f30bc0c8f70 2024-03-23T15:58:56.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.302+0000 7f30c1086640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc1ee200 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f30ac02bf70 tx=0x7f30ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:56.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.306+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac03e040 con 0x7f30bc0c8f70 2024-03-23T15:58:56.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.306+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ac039a40 con 0x7f30bc0c8f70 2024-03-23T15:58:56.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.306+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac039df0 con 0x7f30bc0c8f70 2024-03-23T15:58:56.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.306+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30bc153a00 con 0x7f30bc0c8f70 2024-03-23T15:58:56.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.306+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30bc153fc0 con 0x7f30bc0c8f70 2024-03-23T15:58:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.307+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30bc075740 con 0x7f30bc0c8f70 2024-03-23T15:58:56.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.442+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30ac043390 con 0x7f30bc0c8f70 2024-03-23T15:58:56.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.443+0000 7f30b27fc640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30a006b840 0x7f30a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:56.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.443+0000 7f30c1887640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30a006b840 0x7f30a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:56.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.444+0000 7f30c1887640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30a006b840 0x7f30a006dd00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009aa0 tx=0x7f30b8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:56.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.593+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f30ac07ae70 con 0x7f30bc0c8f70 2024-03-23T15:58:56.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.593+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f30ac077110 con 0x7f30bc0c8f70 2024-03-23T15:58:56.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.827+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30bc0c21f0 con 0x7f30bc0c8f70 2024-03-23T15:58:56.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.877+0000 7f30b27fc640 1 -- 172.21.15.129:0/2186584869 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f30ac034090 con 0x7f30bc0c8f70 2024-03-23T15:58:56.880 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:56.880 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:56.881 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:56.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30a006b840 msgr2=0x7f30a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:56.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30a006b840 0x7f30a006dd00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009aa0 tx=0x7f30b8009190 comp rx=0 tx=0).stop 2024-03-23T15:58:56.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 msgr2=0x7f30bc1ee200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:56.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc1ee200 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f30ac02bf70 tx=0x7f30ac004730 comp rx=0 tx=0).stop 2024-03-23T15:58:56.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 shutdown_connections 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30bc141630 0x7f30bc1400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30bc0c8f70 0x7f30bc1ee200 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30a006b840 0x7f30a006dd00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 --2- 172.21.15.129:0/2186584869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30bc071530 0x7f30bc1f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 >> 172.21.15.129:0/2186584869 conn(0x7f30bc0c8370 msgr2=0x7f30bc13e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 shutdown_connections 2024-03-23T15:58:56.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:56.881+0000 7f30c3aeb640 1 -- 172.21.15.129:0/2186584869 wait complete. 2024-03-23T15:58:57.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:57.499 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:58:58.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:58:58.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.609+0000 7f917aa2e640 1 Processor -- start 2024-03-23T15:58:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.609+0000 7f917aa2e640 1 -- start start 2024-03-23T15:58:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.610+0000 7f917aa2e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f917406c030 0x7f917406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.610+0000 7f917aa2e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f91740cbc60 0x7f917415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.610+0000 7f917aa2e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917406cae0 con 0x7f91740cc080 2024-03-23T15:58:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.610+0000 7f917aa2e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f91740cabc0 con 0x7f91740cbc60 2024-03-23T15:58:58.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.610+0000 7f917aa2e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f91740caea0 con 0x7f917406c030 2024-03-23T15:58:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.610+0000 7f91737fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f91740cbc60 0x7f917415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.611+0000 7f91737fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f91740cbc60 0x7f917415d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46474/0 (socket says 172.21.15.129:46474) 2024-03-23T15:58:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.611+0000 7f91737fe640 1 -- 172.21.15.129:0/262786619 learned_addr learned my addr 172.21.15.129:0/262786619 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:58.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.611+0000 7f9173fff640 1 --2- 172.21.15.129:0/262786619 >> v2:172.21.15.157:3300/0 conn(0x7f917406c030 0x7f917406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:58.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.613+0000 7f91737fe640 1 -- 172.21.15.129:0/262786619 >> v2:172.21.15.157:3300/0 conn(0x7f917406c030 msgr2=0x7f917406c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:58:58.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.613+0000 7f91737fe640 1 --2- 172.21.15.129:0/262786619 >> v2:172.21.15.157:3300/0 conn(0x7f917406c030 0x7f917406c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:58.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.613+0000 7f91737fe640 1 -- 172.21.15.129:0/262786619 >> v1:172.21.15.157:6789/0 conn(0x7f91740cc080 legacy=0x7f917415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:58.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.613+0000 7f91737fe640 1 -- 172.21.15.129:0/262786619 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91740cb180 con 0x7f91740cbc60 2024-03-23T15:58:58.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.615+0000 7f91737fe640 1 --2- 172.21.15.129:0/262786619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cbc60 0x7f917415d900 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f91740cf770 tx=0x7f916802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91305a62b63ce871 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.617+0000 7f9172ffd640 1 -- 172.21.15.129:0/262786619 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168030860 con 0x7f91740cbc60 2024-03-23T15:58:58.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.619+0000 7f9172ffd640 1 -- 172.21.15.129:0/262786619 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91680309c0 con 0x7f91740cbc60 2024-03-23T15:58:58.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.619+0000 7f9172ffd640 1 -- 172.21.15.129:0/262786619 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9168030d50 con 0x7f91740cbc60 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 -- 172.21.15.129:0/262786619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cbc60 msgr2=0x7f917415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 --2- 172.21.15.129:0/262786619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cbc60 0x7f917415d900 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f91740cf770 tx=0x7f916802b9a0 comp rx=0 tx=0).stop 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 -- 172.21.15.129:0/262786619 shutdown_connections 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 --2- 172.21.15.129:0/262786619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cbc60 0x7f917415d900 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 --2- 172.21.15.129:0/262786619 >> v2:172.21.15.157:3300/0 conn(0x7f917406c030 0x7f917406c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 -- 172.21.15.129:0/262786619 >> 172.21.15.129:0/262786619 conn(0x7f91740cfb20 msgr2=0x7f91740cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 -- 172.21.15.129:0/262786619 shutdown_connections 2024-03-23T15:58:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.620+0000 7f917aa2e640 1 -- 172.21.15.129:0/262786619 wait complete. 2024-03-23T15:58:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.621+0000 7f917aa2e640 1 Processor -- start 2024-03-23T15:58:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.621+0000 7f917aa2e640 1 -- start start 2024-03-23T15:58:58.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.622+0000 7f917aa2e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f917406c030 0x7f9174157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:58.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.622+0000 7f917aa2e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91740cbc60 0x7f91740c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:58.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.622+0000 7f917aa2e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 0x7f91741f0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:58.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.622+0000 7f917aa2e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9174155120 con 0x7f91740cc080 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.622+0000 7f917aa2e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f917415de90 con 0x7f91740cbc60 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.622+0000 7f917aa2e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f917415e170 con 0x7f917406c030 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.623+0000 7f9178fcb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 0x7f91741f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.623+0000 7f91737fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91740cbc60 0x7f91740c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.623+0000 7f9178fcb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 0x7f91741f0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46480/0 (socket says 172.21.15.129:46480) 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.623+0000 7f9178fcb640 1 -- 172.21.15.129:0/2876691636 learned_addr learned my addr 172.21.15.129:0/2876691636 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:58:58.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.623+0000 7f9173fff640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f917406c030 0x7f9174157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:58.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.627+0000 7f9178fcb640 1 -- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f917406c030 msgr2=0x7f9174157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:58.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.627+0000 7f9178fcb640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f917406c030 0x7f9174157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:58.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.627+0000 7f9178fcb640 1 -- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91740cbc60 msgr2=0x7f91740c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:58.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.627+0000 7f9178fcb640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91740cbc60 0x7f91740c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:58.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.627+0000 7f9178fcb640 1 -- 172.21.15.129:0/2876691636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91680094d0 con 0x7f91740cc080 2024-03-23T15:58:58.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.629+0000 7f9178fcb640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 0x7f91741f0130 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f916c00c6b0 tx=0x7f916c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:58.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.632+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c003b70 con 0x7f91740cc080 2024-03-23T15:58:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.632+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c003cf0 con 0x7f91740cc080 2024-03-23T15:58:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.632+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917415e450 con 0x7f91740cc080 2024-03-23T15:58:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.632+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c003ea0 con 0x7f91740cc080 2024-03-23T15:58:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.633+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9174157cb0 con 0x7f91740cc080 2024-03-23T15:58:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.633+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9174075740 con 0x7f91740cc080 2024-03-23T15:58:58.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.896+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f916c005310 con 0x7f91740cc080 2024-03-23T15:58:58.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.897+0000 7f9170ff9640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f916006b860 0x7f916006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:58:58.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.897+0000 7f9173fff640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f916006b860 0x7f916006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:58:58.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.898+0000 7f9173fff640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f916006b860 0x7f916006dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9164009b40 tx=0x7f9164009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:58:58.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.923+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f916c059120 con 0x7f91740cc080 2024-03-23T15:58:58.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:58.924+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f916c0553c0 con 0x7f91740cc080 2024-03-23T15:58:59.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.169+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91740d0a30 con 0x7f91740cc080 2024-03-23T15:58:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.209+0000 7f9170ff9640 1 -- 172.21.15.129:0/2876691636 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f916c058a80 con 0x7f91740cc080 2024-03-23T15:58:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:58:59.212 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:58:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:58:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f916006b860 msgr2=0x7f916006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f916006b860 0x7f916006dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9164009b40 tx=0x7f9164009190 comp rx=0 tx=0).stop 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 msgr2=0x7f91741f0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 0x7f91741f0130 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f916c00c6b0 tx=0x7f916c00cb70 comp rx=0 tx=0).stop 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 shutdown_connections 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91740cc080 0x7f91741f0130 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f916006b860 0x7f916006dd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91740cbc60 0x7f91740c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 --2- 172.21.15.129:0/2876691636 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f917406c030 0x7f9174157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 >> 172.21.15.129:0/2876691636 conn(0x7f91740cfb20 msgr2=0x7f91740cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 shutdown_connections 2024-03-23T15:58:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:58:59.212+0000 7f917aa2e640 1 -- 172.21.15.129:0/2876691636 wait complete. 2024-03-23T15:58:59.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:58:59.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:00.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.929+0000 7f179bd7a640 1 Processor -- start 2024-03-23T15:59:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.930+0000 7f179bd7a640 1 -- start start 2024-03-23T15:59:00.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.931+0000 7f179bd7a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17940ccd60 0x7f179415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:00.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.931+0000 7f179bd7a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17940ceb00 0x7f179406c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:00.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.931+0000 7f179bd7a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f179415b290 con 0x7f17940cef40 2024-03-23T15:59:00.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.931+0000 7f179bd7a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f17940ce800 con 0x7f17940ceb00 2024-03-23T15:59:00.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.931+0000 7f179bd7a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1794154a70 con 0x7f17940ccd60 2024-03-23T15:59:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.932+0000 7f1799315640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17940ceb00 0x7f179406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.932+0000 7f1799315640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f17940ceb00 0x7f179406c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59312/0 (socket says 172.21.15.129:59312) 2024-03-23T15:59:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.932+0000 7f1799315640 1 -- 172.21.15.129:0/2087243349 learned_addr learned my addr 172.21.15.129:0/2087243349 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:00.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.932+0000 7f1799b16640 1 --2- 172.21.15.129:0/2087243349 >> v2:172.21.15.157:3300/0 conn(0x7f17940ccd60 0x7f179415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.934+0000 7f1799315640 1 -- 172.21.15.129:0/2087243349 >> v2:172.21.15.157:3300/0 conn(0x7f17940ccd60 msgr2=0x7f179415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.934+0000 7f1799315640 1 --2- 172.21.15.129:0/2087243349 >> v2:172.21.15.157:3300/0 conn(0x7f17940ccd60 0x7f179415ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.934+0000 7f1799315640 1 -- 172.21.15.129:0/2087243349 >> v1:172.21.15.157:6789/0 conn(0x7f17940cef40 legacy=0x7f179415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:00.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.934+0000 7f1799315640 1 -- 172.21.15.129:0/2087243349 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17940cd950 con 0x7f17940ceb00 2024-03-23T15:59:00.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.936+0000 7f1799315640 1 --2- 172.21.15.129:0/2087243349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179406c270 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f179408cf40 tx=0x7f178402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17dae2bfe789627 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:00.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.941+0000 7f1798b14640 1 -- 172.21.15.129:0/2087243349 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178402f850 con 0x7f17940ceb00 2024-03-23T15:59:00.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.941+0000 7f1798b14640 1 -- 172.21.15.129:0/2087243349 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178402f9b0 con 0x7f17940ceb00 2024-03-23T15:59:00.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.941+0000 7f1798b14640 1 -- 172.21.15.129:0/2087243349 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178402fd40 con 0x7f17940ceb00 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 -- 172.21.15.129:0/2087243349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 msgr2=0x7f179406c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 --2- 172.21.15.129:0/2087243349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179406c270 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f179408cf40 tx=0x7f178402ba30 comp rx=0 tx=0).stop 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 -- 172.21.15.129:0/2087243349 shutdown_connections 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 --2- 172.21.15.129:0/2087243349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179406c270 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 --2- 172.21.15.129:0/2087243349 >> v2:172.21.15.157:3300/0 conn(0x7f17940ccd60 0x7f179415ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 -- 172.21.15.129:0/2087243349 >> 172.21.15.129:0/2087243349 conn(0x7f17940b1840 msgr2=0x7f17940b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 -- 172.21.15.129:0/2087243349 shutdown_connections 2024-03-23T15:59:00.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.942+0000 7f179bd7a640 1 -- 172.21.15.129:0/2087243349 wait complete. 2024-03-23T15:59:00.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.943+0000 7f179bd7a640 1 Processor -- start 2024-03-23T15:59:00.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.943+0000 7f179bd7a640 1 -- start start 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179bd7a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17940ccd60 0x7f17941e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179bd7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179bd7a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17940cef40 0x7f17941e4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179bd7a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f179414be30 con 0x7f17940ceb00 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179bd7a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17941f3cd0 con 0x7f17940cef40 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179bd7a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17941f3fb0 con 0x7f17940ccd60 2024-03-23T15:59:00.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f1799315640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f179a317640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17940cef40 0x7f17941e4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f1799315640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59318/0 (socket says 172.21.15.129:59318) 2024-03-23T15:59:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.944+0000 7f1799315640 1 -- 172.21.15.129:0/3622704522 learned_addr learned my addr 172.21.15.129:0/3622704522 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.945+0000 7f1799b16640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17940ccd60 0x7f17941e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.948+0000 7f1799315640 1 -- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17940ccd60 msgr2=0x7f17941e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.948+0000 7f1799315640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17940ccd60 0x7f17941e3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.948+0000 7f1799315640 1 -- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17940cef40 msgr2=0x7f17941e4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.948+0000 7f1799315640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17940cef40 0x7f17941e4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:00.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.948+0000 7f1799315640 1 -- 172.21.15.129:0/3622704522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17840094d0 con 0x7f17940ceb00 2024-03-23T15:59:00.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.950+0000 7f1799315640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179415a670 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f178402ba70 tx=0x7f1784004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:00.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.953+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784037d10 con 0x7f17940ceb00 2024-03-23T15:59:00.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.953+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17941f4290 con 0x7f17940ceb00 2024-03-23T15:59:00.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.953+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17941f4850 con 0x7f17940ceb00 2024-03-23T15:59:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.954+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17941f4a30 con 0x7f17940ceb00 2024-03-23T15:59:00.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.955+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1784037e70 con 0x7f17940ceb00 2024-03-23T15:59:00.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:00.955+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178404b660 con 0x7f17940ceb00 2024-03-23T15:59:01.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.237+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f178404b7c0 con 0x7f17940ceb00 2024-03-23T15:59:01.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.237+0000 7f178a7fc640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f177806b840 0x7f177806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:01.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.237+0000 7f1799b16640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f177806b840 0x7f177806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:01.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.238+0000 7f1799b16640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f177806b840 0x7f177806dd00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f17900079a0 tx=0x7f1790008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:01.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.264+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f178403c070 con 0x7f17940ceb00 2024-03-23T15:59:01.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.264+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f17840785c0 con 0x7f17940ceb00 2024-03-23T15:59:01.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.490+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f179404f530 con 0x7f17940ceb00 2024-03-23T15:59:01.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.504+0000 7f178a7fc640 1 -- 172.21.15.129:0/3622704522 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f178407bc80 con 0x7f17940ceb00 2024-03-23T15:59:01.507 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:01.507 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:01.507 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f177806b840 msgr2=0x7f177806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f177806b840 0x7f177806dd00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f17900079a0 tx=0x7f1790008040 comp rx=0 tx=0).stop 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 msgr2=0x7f179415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179415a670 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f178402ba70 tx=0x7f1784004730 comp rx=0 tx=0).stop 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 shutdown_connections 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f177806b840 0x7f177806dd00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17940cef40 0x7f17941e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17940ceb00 0x7f179415a670 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 --2- 172.21.15.129:0/3622704522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17940ccd60 0x7f17941e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 >> 172.21.15.129:0/3622704522 conn(0x7f17940b1840 msgr2=0x7f17941540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 shutdown_connections 2024-03-23T15:59:01.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:01.506+0000 7f179bd7a640 1 -- 172.21.15.129:0/3622704522 wait complete. 2024-03-23T15:59:02.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:02.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:03.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:03.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.227+0000 7f37f742e640 1 Processor -- start 2024-03-23T15:59:03.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.227+0000 7f37f742e640 1 -- start start 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.228+0000 7f37f742e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f37f01520b0 0x7f37f0158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.228+0000 7f37f742e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f37f006f950 0x7f37f0159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.228+0000 7f37f742e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f015b5d0 con 0x7f37f00c8290 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.228+0000 7f37f742e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f37f015c430 con 0x7f37f006f950 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.228+0000 7f37f742e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f37f006fd60 con 0x7f37f01520b0 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.229+0000 7f37f51ca640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f37f01520b0 0x7f37f0158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.229+0000 7f37f59cb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f37f006f950 0x7f37f0159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.229+0000 7f37f49c9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f37f00c8290 0x7f37f00cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36520/0 (socket says 172.21.15.129:36520) 2024-03-23T15:59:03.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.229+0000 7f37f49c9640 1 -- 172.21.15.129:0/3617738368 learned_addr learned my addr 172.21.15.129:0/3617738368 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:03.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.232+0000 7f37f51ca640 1 -- 172.21.15.129:0/3617738368 >> v2:172.21.15.129:3300/0 conn(0x7f37f006f950 msgr2=0x7f37f0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:03.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.232+0000 7f37f51ca640 1 --2- 172.21.15.129:0/3617738368 >> v2:172.21.15.129:3300/0 conn(0x7f37f006f950 0x7f37f0159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.232+0000 7f37f51ca640 1 -- 172.21.15.129:0/3617738368 >> v1:172.21.15.129:6789/0 conn(0x7f37f00c8290 legacy=0x7f37f00cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:03.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.232+0000 7f37f51ca640 1 -- 172.21.15.129:0/3617738368 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f0152490 con 0x7f37f01520b0 2024-03-23T15:59:03.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.235+0000 7f37f51ca640 1 --2- 172.21.15.129:0/3617738368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f01520b0 0x7f37f0158a60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f37f00ce620 tx=0x7f37ec02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90cc86e484284b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:03.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.240+0000 7f37e7fff640 1 -- 172.21.15.129:0/3617738368 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec030610 con 0x7f37f01520b0 2024-03-23T15:59:03.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.240+0000 7f37e7fff640 1 -- 172.21.15.129:0/3617738368 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ec030770 con 0x7f37f01520b0 2024-03-23T15:59:03.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.240+0000 7f37e7fff640 1 -- 172.21.15.129:0/3617738368 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec030b00 con 0x7f37f01520b0 2024-03-23T15:59:03.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 -- 172.21.15.129:0/3617738368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f01520b0 msgr2=0x7f37f0158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:03.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 --2- 172.21.15.129:0/3617738368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f01520b0 0x7f37f0158a60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f37f00ce620 tx=0x7f37ec02b960 comp rx=0 tx=0).stop 2024-03-23T15:59:03.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 -- 172.21.15.129:0/3617738368 shutdown_connections 2024-03-23T15:59:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 --2- 172.21.15.129:0/3617738368 >> v2:172.21.15.129:3300/0 conn(0x7f37f006f950 0x7f37f0159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 --2- 172.21.15.129:0/3617738368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f01520b0 0x7f37f0158a60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 -- 172.21.15.129:0/3617738368 >> 172.21.15.129:0/3617738368 conn(0x7f37f0061b50 msgr2=0x7f37f0061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 -- 172.21.15.129:0/3617738368 shutdown_connections 2024-03-23T15:59:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.241+0000 7f37f742e640 1 -- 172.21.15.129:0/3617738368 wait complete. 2024-03-23T15:59:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.243+0000 7f37f742e640 1 Processor -- start 2024-03-23T15:59:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.243+0000 7f37f742e640 1 -- start start 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.244+0000 7f37f742e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 0x7f37f0152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.244+0000 7f37f742e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f00c8290 0x7f37f00ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.244+0000 7f37f742e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37f01520b0 0x7f37f0157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.244+0000 7f37f742e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0155ca0 con 0x7f37f006f950 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.244+0000 7f37f742e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0155f80 con 0x7f37f00c8290 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.244+0000 7f37f742e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f37f0156260 con 0x7f37f01520b0 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.245+0000 7f37f51ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 0x7f37f0152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.245+0000 7f37f49c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f00c8290 0x7f37f00ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.245+0000 7f37f51ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 0x7f37f0152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59332/0 (socket says 172.21.15.129:59332) 2024-03-23T15:59:03.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.245+0000 7f37f51ca640 1 -- 172.21.15.129:0/2368051313 learned_addr learned my addr 172.21.15.129:0/2368051313 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:03.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.245+0000 7f37f59cb640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37f01520b0 0x7f37f0157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:03.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.249+0000 7f37f51ca640 1 -- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37f01520b0 msgr2=0x7f37f0157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:03.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.249+0000 7f37f51ca640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37f01520b0 0x7f37f0157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.249+0000 7f37f51ca640 1 -- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f00c8290 msgr2=0x7f37f00ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:03.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.249+0000 7f37f51ca640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f00c8290 0x7f37f00ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.249+0000 7f37f51ca640 1 -- 172.21.15.129:0/2368051313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec009400 con 0x7f37f006f950 2024-03-23T15:59:03.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.251+0000 7f37f51ca640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 0x7f37f0152d70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009890 tx=0x7f37ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:03.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.254+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec03e070 con 0x7f37f006f950 2024-03-23T15:59:03.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.254+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f0156540 con 0x7f37f006f950 2024-03-23T15:59:03.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.254+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f0156b00 con 0x7f37f006f950 2024-03-23T15:59:03.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.254+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ec039910 con 0x7f37f006f950 2024-03-23T15:59:03.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.254+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec039c80 con 0x7f37f006f950 2024-03-23T15:59:03.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.255+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d0005290 con 0x7f37f006f950 2024-03-23T15:59:03.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.290+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37ec04d440 con 0x7f37f006f950 2024-03-23T15:59:03.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.290+0000 7f37e5ffb640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f37d406b840 0x7f37d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:03.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.291+0000 7f37f49c9640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f37d406b840 0x7f37d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:03.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.291+0000 7f37f49c9640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f37d406b840 0x7f37d406dd00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37f014e5d0 tx=0x7f37e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:03.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.296+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f37ec07aac0 con 0x7f37f006f950 2024-03-23T15:59:03.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.560+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37ec076d60 con 0x7f37f006f950 2024-03-23T15:59:03.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.793+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37d00058f0 con 0x7f37f006f950 2024-03-23T15:59:03.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.806+0000 7f37e5ffb640 1 -- 172.21.15.129:0/2368051313 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f37d00058f0 con 0x7f37f006f950 2024-03-23T15:59:03.808 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:03.809 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:03.809 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:03.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f37d406b840 msgr2=0x7f37d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f37d406b840 0x7f37d406dd00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37f014e5d0 tx=0x7f37e0009040 comp rx=0 tx=0).stop 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 msgr2=0x7f37f0152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 0x7f37f0152d70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009890 tx=0x7f37ec0046c0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 shutdown_connections 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37f01520b0 0x7f37f0157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f37d406b840 0x7f37d406dd00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37f00c8290 0x7f37f00ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 --2- 172.21.15.129:0/2368051313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37f006f950 0x7f37f0152d70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 >> 172.21.15.129:0/2368051313 conn(0x7f37f0061b50 msgr2=0x7f37f01505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 shutdown_connections 2024-03-23T15:59:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:03.809+0000 7f37f742e640 1 -- 172.21.15.129:0/2368051313 wait complete. 2024-03-23T15:59:04.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:04.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:05.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:05.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.528+0000 7f5492b19640 1 Processor -- start 2024-03-23T15:59:05.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.528+0000 7f5492b19640 1 -- start start 2024-03-23T15:59:05.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.529+0000 7f5492b19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0ccd60 0x7f548c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:05.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.529+0000 7f5492b19640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f548c0cef40 0x7f548c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:05.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.529+0000 7f5492b19640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f548c15b290 con 0x7f548c0ceb00 2024-03-23T15:59:05.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.529+0000 7f5492b19640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f548c0ce800 con 0x7f548c0cef40 2024-03-23T15:59:05.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.529+0000 7f5492b19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f548c154a70 con 0x7f548c0ccd60 2024-03-23T15:59:05.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.530+0000 7f54910b6640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f548c0cef40 0x7f548c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:05.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.530+0000 7f54908b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0ccd60 0x7f548c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:05.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.530+0000 7f548bfff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f548c0ceb00 0x7f548c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36536/0 (socket says 172.21.15.129:36536) 2024-03-23T15:59:05.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.530+0000 7f548bfff640 1 -- 172.21.15.129:0/3333910350 learned_addr learned my addr 172.21.15.129:0/3333910350 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:05.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.533+0000 7f54908b5640 1 -- 172.21.15.129:0/3333910350 >> v2:172.21.15.129:3300/0 conn(0x7f548c0cef40 msgr2=0x7f548c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:05.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.533+0000 7f54908b5640 1 --2- 172.21.15.129:0/3333910350 >> v2:172.21.15.129:3300/0 conn(0x7f548c0cef40 0x7f548c15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:05.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.533+0000 7f54908b5640 1 -- 172.21.15.129:0/3333910350 >> v1:172.21.15.129:6789/0 conn(0x7f548c0ceb00 legacy=0x7f548c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:05.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.533+0000 7f54908b5640 1 -- 172.21.15.129:0/3333910350 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f548c0cd950 con 0x7f548c0ccd60 2024-03-23T15:59:05.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.535+0000 7f54908b5640 1 --2- 172.21.15.129:0/3333910350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0ccd60 0x7f548c15ad50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f54800097a0 tx=0x7f548002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e236739940e86f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.541+0000 7f548b7fe640 1 -- 172.21.15.129:0/3333910350 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480030610 con 0x7f548c0ccd60 2024-03-23T15:59:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.541+0000 7f548b7fe640 1 -- 172.21.15.129:0/3333910350 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5480030770 con 0x7f548c0ccd60 2024-03-23T15:59:05.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.541+0000 7f548b7fe640 1 -- 172.21.15.129:0/3333910350 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480030b00 con 0x7f548c0ccd60 2024-03-23T15:59:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 -- 172.21.15.129:0/3333910350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0ccd60 msgr2=0x7f548c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 --2- 172.21.15.129:0/3333910350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0ccd60 0x7f548c15ad50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f54800097a0 tx=0x7f548002b960 comp rx=0 tx=0).stop 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 -- 172.21.15.129:0/3333910350 shutdown_connections 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 --2- 172.21.15.129:0/3333910350 >> v2:172.21.15.129:3300/0 conn(0x7f548c0cef40 0x7f548c15b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 --2- 172.21.15.129:0/3333910350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0ccd60 0x7f548c15ad50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 -- 172.21.15.129:0/3333910350 >> 172.21.15.129:0/3333910350 conn(0x7f548c0b1840 msgr2=0x7f548c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 -- 172.21.15.129:0/3333910350 shutdown_connections 2024-03-23T15:59:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.542+0000 7f5492b19640 1 -- 172.21.15.129:0/3333910350 wait complete. 2024-03-23T15:59:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.544+0000 7f5492b19640 1 Processor -- start 2024-03-23T15:59:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.544+0000 7f5492b19640 1 -- start start 2024-03-23T15:59:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.545+0000 7f5492b19640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f548c0ccd60 0x7f548c157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.545+0000 7f5492b19640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 0x7f548c1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.545+0000 7f5492b19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0cef40 0x7f548c1e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.545+0000 7f5492b19640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f548c14bc10 con 0x7f548c0ceb00 2024-03-23T15:59:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.545+0000 7f5492b19640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f548c0c9390 con 0x7f548c0ccd60 2024-03-23T15:59:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.545+0000 7f5492b19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f548c0c7280 con 0x7f548c0cef40 2024-03-23T15:59:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.546+0000 7f548bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 0x7f548c1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.546+0000 7f54910b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0cef40 0x7f548c1e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.546+0000 7f54908b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f548c0ccd60 0x7f548c157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.546+0000 7f548bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 0x7f548c1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59350/0 (socket says 172.21.15.129:59350) 2024-03-23T15:59:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.546+0000 7f548bfff640 1 -- 172.21.15.129:0/2391957759 learned_addr learned my addr 172.21.15.129:0/2391957759 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:05.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.549+0000 7f548bfff640 1 -- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0cef40 msgr2=0x7f548c1e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:05.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.549+0000 7f548bfff640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0cef40 0x7f548c1e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:05.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.549+0000 7f548bfff640 1 -- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f548c0ccd60 msgr2=0x7f548c157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:05.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.549+0000 7f548bfff640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f548c0ccd60 0x7f548c157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:05.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.549+0000 7f548bfff640 1 -- 172.21.15.129:0/2391957759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5480009400 con 0x7f548c0ceb00 2024-03-23T15:59:05.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.553+0000 7f548bfff640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 0x7f548c1f0d00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f547800c430 tx=0x7f547800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:05.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.560+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478005ce0 con 0x7f548c0ceb00 2024-03-23T15:59:05.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.560+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5478005e40 con 0x7f548c0ceb00 2024-03-23T15:59:05.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.560+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54780072f0 con 0x7f548c0ceb00 2024-03-23T15:59:05.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.560+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548c0c7560 con 0x7f548c0ceb00 2024-03-23T15:59:05.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.560+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f548c1576d0 con 0x7f548c0ceb00 2024-03-23T15:59:05.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.561+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f548c04f530 con 0x7f548c0ceb00 2024-03-23T15:59:05.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.814+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5478007510 con 0x7f548c0ceb00 2024-03-23T15:59:05.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.815+0000 7f54897fa640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f547006b860 0x7f547006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:05.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.815+0000 7f54908b5640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f547006b860 0x7f547006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:05.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.816+0000 7f54908b5640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f547006b860 0x7f547006dd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5480036000 tx=0x7f5480008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:05.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.837+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5478058270 con 0x7f548c0ceb00 2024-03-23T15:59:05.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:05.837+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5478054510 con 0x7f548c0ceb00 2024-03-23T15:59:06.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.087+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f548c0686f0 con 0x7f548c0ceb00 2024-03-23T15:59:06.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.108+0000 7f54897fa640 1 -- 172.21.15.129:0/2391957759 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5478017070 con 0x7f548c0ceb00 2024-03-23T15:59:06.111 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:06.111 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:06.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f547006b860 msgr2=0x7f547006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:06.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f547006b860 0x7f547006dd20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5480036000 tx=0x7f5480008030 comp rx=0 tx=0).stop 2024-03-23T15:59:06.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 msgr2=0x7f548c1f0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 0x7f548c1f0d00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f547800c430 tx=0x7f547800c8f0 comp rx=0 tx=0).stop 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 shutdown_connections 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f548c0cef40 0x7f548c1e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0ceb00 0x7f548c1f0d00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f547006b860 0x7f547006dd20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 --2- 172.21.15.129:0/2391957759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f548c0ccd60 0x7f548c157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 >> 172.21.15.129:0/2391957759 conn(0x7f548c0b1840 msgr2=0x7f548c153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 shutdown_connections 2024-03-23T15:59:06.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:06.111+0000 7f5492b19640 1 -- 172.21.15.129:0/2391957759 wait complete. 2024-03-23T15:59:06.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:06.731 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:07.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:07.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.833+0000 7f241a23b640 1 Processor -- start 2024-03-23T15:59:07.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.833+0000 7f241a23b640 1 -- start start 2024-03-23T15:59:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.834+0000 7f241a23b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.834+0000 7f241a23b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24141591e0 con 0x7f24140c8f70 2024-03-23T15:59:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.834+0000 7f241a23b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2414141a10 con 0x7f2414071530 2024-03-23T15:59:07.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.834+0000 7f241a23b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f241415c740 con 0x7f2414141630 2024-03-23T15:59:07.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.835+0000 7f2412ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f24140c8f70 0x7f241415c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:36550/0 (socket says 172.21.15.129:36550) 2024-03-23T15:59:07.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.835+0000 7f2412ffd640 1 -- 172.21.15.129:0/2379536442 learned_addr learned my addr 172.21.15.129:0/2379536442 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:07.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.835+0000 7f24137fe640 1 --2- 172.21.15.129:0/2379536442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:07.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.837+0000 7f24127fc640 1 -- 172.21.15.129:0/2379536442 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523451827 0 0) 0x7f2408001110 con 0x7f24140c8f70 2024-03-23T15:59:07.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.837+0000 7f24127fc640 1 -- 172.21.15.129:0/2379536442 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23fc003520 con 0x7f24140c8f70 2024-03-23T15:59:07.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.838+0000 7f24137fe640 1 -- 172.21.15.129:0/2379536442 >> v1:172.21.15.157:6789/0 conn(0x7f2414071530 legacy=0x7f24140c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:07.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.838+0000 7f24137fe640 1 -- 172.21.15.129:0/2379536442 >> v1:172.21.15.129:6789/0 conn(0x7f24140c8f70 legacy=0x7f241415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:07.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.838+0000 7f24137fe640 1 -- 172.21.15.129:0/2379536442 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241415bba0 con 0x7f2414141630 2024-03-23T15:59:07.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.840+0000 7f24137fe640 1 --2- 172.21.15.129:0/2379536442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414158ca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f24040097d0 tx=0x7f240402b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a169958fb0337ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.845+0000 7f24127fc640 1 -- 172.21.15.129:0/2379536442 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404030890 con 0x7f2414141630 2024-03-23T15:59:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.845+0000 7f24127fc640 1 -- 172.21.15.129:0/2379536442 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24040309f0 con 0x7f2414141630 2024-03-23T15:59:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.845+0000 7f24127fc640 1 -- 172.21.15.129:0/2379536442 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404030d80 con 0x7f2414141630 2024-03-23T15:59:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 -- 172.21.15.129:0/2379536442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 msgr2=0x7f2414158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 --2- 172.21.15.129:0/2379536442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414158ca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f24040097d0 tx=0x7f240402b9d0 comp rx=0 tx=0).stop 2024-03-23T15:59:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 -- 172.21.15.129:0/2379536442 shutdown_connections 2024-03-23T15:59:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 --2- 172.21.15.129:0/2379536442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414158ca0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 -- 172.21.15.129:0/2379536442 >> 172.21.15.129:0/2379536442 conn(0x7f24140c8370 msgr2=0x7f24140c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 -- 172.21.15.129:0/2379536442 shutdown_connections 2024-03-23T15:59:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.846+0000 7f241a23b640 1 -- 172.21.15.129:0/2379536442 wait complete. 2024-03-23T15:59:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.847+0000 7f241a23b640 1 Processor -- start 2024-03-23T15:59:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.847+0000 7f241a23b640 1 -- start start 2024-03-23T15:59:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f241a23b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 0x7f24140c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f241a23b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24140c8f70 0x7f24141f1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f241a23b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414140450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f241a23b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f24140c3ab0 con 0x7f2414071530 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f241a23b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f24141f0ba0 con 0x7f24140c8f70 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f241a23b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24140c2ea0 con 0x7f2414141630 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f24137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 0x7f24140c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.848+0000 7f2413fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.849+0000 7f2412ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24140c8f70 0x7f24141f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.849+0000 7f24137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 0x7f24140c3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59354/0 (socket says 172.21.15.129:59354) 2024-03-23T15:59:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.849+0000 7f24137fe640 1 -- 172.21.15.129:0/1710581912 learned_addr learned my addr 172.21.15.129:0/1710581912 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.852+0000 7f24137fe640 1 -- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 msgr2=0x7f2414140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.852+0000 7f24137fe640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.852+0000 7f24137fe640 1 -- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24140c8f70 msgr2=0x7f24141f1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.852+0000 7f24137fe640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24140c8f70 0x7f24141f1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:07.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.852+0000 7f24137fe640 1 -- 172.21.15.129:0/1710581912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2404009400 con 0x7f2414071530 2024-03-23T15:59:07.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.854+0000 7f24137fe640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 0x7f24140c3f40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f24040097d0 tx=0x7f24040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:07.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.858+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240403e070 con 0x7f2414071530 2024-03-23T15:59:07.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.858+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2404039a10 con 0x7f2414071530 2024-03-23T15:59:07.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.858+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24140c3180 con 0x7f2414071530 2024-03-23T15:59:07.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.858+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404039dc0 con 0x7f2414071530 2024-03-23T15:59:07.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.858+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24140c3740 con 0x7f2414071530 2024-03-23T15:59:07.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.859+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2414075740 con 0x7f2414071530 2024-03-23T15:59:07.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.877+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f24040423f0 con 0x7f2414071530 2024-03-23T15:59:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.878+0000 7f23fbfff640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f240006b840 0x7f240006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:07.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.878+0000 7f2412ffd640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f240006b840 0x7f240006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:07.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:07.879+0000 7f2412ffd640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f240006b840 0x7f240006dd00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f24140c2770 tx=0x7f2408007bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:08.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.126+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f240407b220 con 0x7f2414071530 2024-03-23T15:59:08.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.126+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f24040774c0 con 0x7f2414071530 2024-03-23T15:59:08.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.346+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f241413db50 con 0x7f2414071530 2024-03-23T15:59:08.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.361+0000 7f23fbfff640 1 -- 172.21.15.129:0/1710581912 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f2404048070 con 0x7f2414071530 2024-03-23T15:59:08.363 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:08.363 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:08.364 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f240006b840 msgr2=0x7f240006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f240006b840 0x7f240006dd00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f24140c2770 tx=0x7f2408007bb0 comp rx=0 tx=0).stop 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 msgr2=0x7f24140c3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 0x7f24140c3f40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f24040097d0 tx=0x7f24040046c0 comp rx=0 tx=0).stop 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 shutdown_connections 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2414141630 0x7f2414140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f240006b840 0x7f240006dd00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24140c8f70 0x7f24141f1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 --2- 172.21.15.129:0/1710581912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2414071530 0x7f24140c3f40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 >> 172.21.15.129:0/1710581912 conn(0x7f24140c8370 msgr2=0x7f24140c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 shutdown_connections 2024-03-23T15:59:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:08.364+0000 7f241a23b640 1 -- 172.21.15.129:0/1710581912 wait complete. 2024-03-23T15:59:08.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:59:08.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:59:08.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:08.984 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:09.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.085+0000 7f6b48d9a640 1 Processor -- start 2024-03-23T15:59:10.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.085+0000 7f6b48d9a640 1 -- start start 2024-03-23T15:59:10.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.086+0000 7f6b48d9a640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6b4415dba0 0x7f6b4406cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:10.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.086+0000 7f6b48d9a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b44070640 con 0x7f6b440cca30 2024-03-23T15:59:10.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.086+0000 7f6b48d9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b440c4c00 con 0x7f6b4415df80 2024-03-23T15:59:10.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.086+0000 7f6b48d9a640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6b440ccea0 con 0x7f6b4415dba0 2024-03-23T15:59:10.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.087+0000 7f6b42d9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6b4415df80 0x7f6b441540a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:36994/0 (socket says 172.21.15.129:36994) 2024-03-23T15:59:10.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.087+0000 7f6b42d9d640 1 -- 172.21.15.129:0/2196674649 learned_addr learned my addr 172.21.15.129:0/2196674649 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:10.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.087+0000 7f6b4259c640 1 --2- 172.21.15.129:0/2196674649 >> v2:172.21.15.129:3300/0 conn(0x7f6b4415dba0 0x7f6b4406cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:10.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.090+0000 7f6b4159a640 1 -- 172.21.15.129:0/2196674649 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3492625368 0 0) 0x7f6b3c001490 con 0x7f6b4415df80 2024-03-23T15:59:10.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.090+0000 7f6b4159a640 1 -- 172.21.15.129:0/2196674649 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2c003520 con 0x7f6b4415df80 2024-03-23T15:59:10.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.092+0000 7f6b4259c640 1 -- 172.21.15.129:0/2196674649 >> v1:172.21.15.157:6789/0 conn(0x7f6b4415df80 legacy=0x7f6b441540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.092+0000 7f6b4259c640 1 -- 172.21.15.129:0/2196674649 >> v1:172.21.15.129:6789/0 conn(0x7f6b440cca30 legacy=0x7f6b4406ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.092+0000 7f6b4259c640 1 -- 172.21.15.129:0/2196674649 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4406c240 con 0x7f6b4415dba0 2024-03-23T15:59:10.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.096+0000 7f6b4259c640 1 --2- 172.21.15.129:0/2196674649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b4415dba0 0x7f6b4406cf70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6b340097d0 tx=0x7f6b3402b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c947003f85dcd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:10.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.100+0000 7f6b4159a640 1 -- 172.21.15.129:0/2196674649 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34030890 con 0x7f6b4415dba0 2024-03-23T15:59:10.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.100+0000 7f6b4159a640 1 -- 172.21.15.129:0/2196674649 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b340309f0 con 0x7f6b4415dba0 2024-03-23T15:59:10.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.100+0000 7f6b4159a640 1 -- 172.21.15.129:0/2196674649 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34030d80 con 0x7f6b4415dba0 2024-03-23T15:59:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 -- 172.21.15.129:0/2196674649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b4415dba0 msgr2=0x7f6b4406cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/2196674649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b4415dba0 0x7f6b4406cf70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6b340097d0 tx=0x7f6b3402b9d0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 -- 172.21.15.129:0/2196674649 shutdown_connections 2024-03-23T15:59:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/2196674649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b4415dba0 0x7f6b4406cf70 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 -- 172.21.15.129:0/2196674649 >> 172.21.15.129:0/2196674649 conn(0x7f6b44071300 msgr2=0x7f6b440716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 -- 172.21.15.129:0/2196674649 shutdown_connections 2024-03-23T15:59:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.101+0000 7f6b48d9a640 1 -- 172.21.15.129:0/2196674649 wait complete. 2024-03-23T15:59:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.102+0000 7f6b48d9a640 1 Processor -- start 2024-03-23T15:59:10.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 -- start start 2024-03-23T15:59:10.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 0x7f6b441e5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:10.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b4415dba0 0x7f6b441e4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:10.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b4415df80 0x7f6b440c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:10.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441e3f50 con 0x7f6b440cca30 2024-03-23T15:59:10.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441e53e0 con 0x7f6b4415df80 2024-03-23T15:59:10.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.103+0000 7f6b48d9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6b4415a910 con 0x7f6b4415dba0 2024-03-23T15:59:10.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.104+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 0x7f6b441e5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:10.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.104+0000 7f6b42d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b4415df80 0x7f6b440c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:10.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.104+0000 7f6b41d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b4415dba0 0x7f6b441e4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:10.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.104+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 0x7f6b441e5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59364/0 (socket says 172.21.15.129:59364) 2024-03-23T15:59:10.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.104+0000 7f6b4259c640 1 -- 172.21.15.129:0/1392622869 learned_addr learned my addr 172.21.15.129:0/1392622869 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:10.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.107+0000 7f6b4259c640 1 -- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b4415dba0 msgr2=0x7f6b441e4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.107+0000 7f6b4259c640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b4415dba0 0x7f6b441e4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.107+0000 7f6b4259c640 1 -- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b4415df80 msgr2=0x7f6b440c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.107+0000 7f6b4259c640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b4415df80 0x7f6b440c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.107+0000 7f6b4259c640 1 -- 172.21.15.129:0/1392622869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b34009400 con 0x7f6b440cca30 2024-03-23T15:59:10.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.110+0000 7f6b4259c640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 0x7f6b441e5660 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6b340098c0 tx=0x7f6b34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.113+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3403e040 con 0x7f6b440cca30 2024-03-23T15:59:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.113+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b34039df0 con 0x7f6b440cca30 2024-03-23T15:59:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.113+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b441e3920 con 0x7f6b440cca30 2024-03-23T15:59:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.113+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34043680 con 0x7f6b440cca30 2024-03-23T15:59:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.113+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4414cee0 con 0x7f6b440cca30 2024-03-23T15:59:10.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.114+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b44075740 con 0x7f6b440cca30 2024-03-23T15:59:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.133+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6b340423e0 con 0x7f6b440cca30 2024-03-23T15:59:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.133+0000 7f6b32ffd640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b2406b810 0x7f6b2406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.134+0000 7f6b41d9b640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b2406b810 0x7f6b2406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:10.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.134+0000 7f6b41d9b640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b2406b810 0x7f6b2406dcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6b440ce780 tx=0x7f6b38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:10.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.385+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6b3407b350 con 0x7f6b440cca30 2024-03-23T15:59:10.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.386+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6b340775f0 con 0x7f6b440cca30 2024-03-23T15:59:10.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.622+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b440c99f0 con 0x7f6b440cca30 2024-03-23T15:59:10.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.643+0000 7f6b32ffd640 1 -- 172.21.15.129:0/1392622869 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6b3407acb0 con 0x7f6b440cca30 2024-03-23T15:59:10.646 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:10.646 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:10.646 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b2406b810 msgr2=0x7f6b2406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b2406b810 0x7f6b2406dcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6b440ce780 tx=0x7f6b38009040 comp rx=0 tx=0).stop 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 msgr2=0x7f6b441e5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 0x7f6b441e5660 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6b340098c0 tx=0x7f6b34004730 comp rx=0 tx=0).stop 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 shutdown_connections 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6b2406b810 0x7f6b2406dcd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b4415df80 0x7f6b440c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b4415dba0 0x7f6b441e4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 --2- 172.21.15.129:0/1392622869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b440cca30 0x7f6b441e5660 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 >> 172.21.15.129:0/1392622869 conn(0x7f6b44071300 msgr2=0x7f6b440707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:10.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 shutdown_connections 2024-03-23T15:59:10.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:10.646+0000 7f6b48d9a640 1 -- 172.21.15.129:0/1392622869 wait complete. 2024-03-23T15:59:11.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:11.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:12.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.365+0000 7f88737af640 1 Processor -- start 2024-03-23T15:59:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.365+0000 7f88737af640 1 -- start start 2024-03-23T15:59:12.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.366+0000 7f88737af640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f886c15dba0 0x7f886c06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:12.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.366+0000 7f88737af640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c070640 con 0x7f886c15df80 2024-03-23T15:59:12.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.366+0000 7f88737af640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c0c4c00 con 0x7f886c0cca30 2024-03-23T15:59:12.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.366+0000 7f88737af640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f886c0ccea0 con 0x7f886c15dba0 2024-03-23T15:59:12.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.366+0000 7f887154b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f886c15dba0 0x7f886c06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:12.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.366+0000 7f8870d4a640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f886c0cca30 0x7f886c06ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:48776/0 (socket says 172.21.15.129:48776) 2024-03-23T15:59:12.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.367+0000 7f8870d4a640 1 -- 172.21.15.129:0/2922421904 learned_addr learned my addr 172.21.15.129:0/2922421904 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:12.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.370+0000 7f8863fff640 1 -- 172.21.15.129:0/2922421904 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 227623939 0 0) 0x7f885c001160 con 0x7f886c0cca30 2024-03-23T15:59:12.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.370+0000 7f8863fff640 1 -- 172.21.15.129:0/2922421904 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8858003520 con 0x7f886c0cca30 2024-03-23T15:59:12.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.371+0000 7f887154b640 1 -- 172.21.15.129:0/2922421904 >> v1:172.21.15.157:6789/0 conn(0x7f886c0cca30 legacy=0x7f886c06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:12.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.371+0000 7f887154b640 1 -- 172.21.15.129:0/2922421904 >> v1:172.21.15.129:6789/0 conn(0x7f886c15df80 legacy=0x7f886c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:12.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.371+0000 7f887154b640 1 -- 172.21.15.129:0/2922421904 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c06c240 con 0x7f886c15dba0 2024-03-23T15:59:12.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.374+0000 7f887154b640 1 --2- 172.21.15.129:0/2922421904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c06cf70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f88680097d0 tx=0x7f886802b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f55f2c50a14a5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:12.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.379+0000 7f8863fff640 1 -- 172.21.15.129:0/2922421904 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868030890 con 0x7f886c15dba0 2024-03-23T15:59:12.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.379+0000 7f8863fff640 1 -- 172.21.15.129:0/2922421904 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88680309f0 con 0x7f886c15dba0 2024-03-23T15:59:12.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.379+0000 7f8863fff640 1 -- 172.21.15.129:0/2922421904 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868030d80 con 0x7f886c15dba0 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 -- 172.21.15.129:0/2922421904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 msgr2=0x7f886c06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 --2- 172.21.15.129:0/2922421904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c06cf70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f88680097d0 tx=0x7f886802b9d0 comp rx=0 tx=0).stop 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 -- 172.21.15.129:0/2922421904 shutdown_connections 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 --2- 172.21.15.129:0/2922421904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c06cf70 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 -- 172.21.15.129:0/2922421904 >> 172.21.15.129:0/2922421904 conn(0x7f886c071300 msgr2=0x7f886c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 -- 172.21.15.129:0/2922421904 shutdown_connections 2024-03-23T15:59:12.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.380+0000 7f88737af640 1 -- 172.21.15.129:0/2922421904 wait complete. 2024-03-23T15:59:12.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.382+0000 7f88737af640 1 Processor -- start 2024-03-23T15:59:12.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.382+0000 7f88737af640 1 -- start start 2024-03-23T15:59:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.383+0000 7f88737af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 0x7f886c0c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.383+0000 7f88737af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c1492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.383+0000 7f88737af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c15df80 0x7f886c1497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.383+0000 7f88737af640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f886c0c7e40 con 0x7f886c15dba0 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.383+0000 7f88737af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f886c0c7630 con 0x7f886c15df80 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.383+0000 7f88737af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f886c0708d0 con 0x7f886c0cca30 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.384+0000 7f887154b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 0x7f886c0c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.384+0000 7f8871d4c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c15df80 0x7f886c1497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.384+0000 7f887154b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 0x7f886c0c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:58824/0 (socket says 172.21.15.129:58824) 2024-03-23T15:59:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.384+0000 7f887154b640 1 -- 172.21.15.129:0/3755068211 learned_addr learned my addr 172.21.15.129:0/3755068211 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:12.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.384+0000 7f8870d4a640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c1492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:12.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.388+0000 7f887154b640 1 -- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c15df80 msgr2=0x7f886c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:12.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.388+0000 7f887154b640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c15df80 0x7f886c1497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:12.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.388+0000 7f887154b640 1 -- 172.21.15.129:0/3755068211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 msgr2=0x7f886c1492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:12.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.388+0000 7f887154b640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c1492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:12.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.388+0000 7f887154b640 1 -- 172.21.15.129:0/3755068211 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8868009400 con 0x7f886c0cca30 2024-03-23T15:59:12.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.390+0000 7f887154b640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 0x7f886c0c88a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f88680098f0 tx=0x7f8868004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:12.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.393+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886803e040 con 0x7f886c0cca30 2024-03-23T15:59:12.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.393+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c1f4500 con 0x7f886c0cca30 2024-03-23T15:59:12.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.393+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c0c7880 con 0x7f886c0cca30 2024-03-23T15:59:12.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.394+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886c075740 con 0x7f886c0cca30 2024-03-23T15:59:12.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.395+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8868039d50 con 0x7f886c0cca30 2024-03-23T15:59:12.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.395+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886804e640 con 0x7f886c0cca30 2024-03-23T15:59:12.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.705+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f88680443f0 con 0x7f886c0cca30 2024-03-23T15:59:12.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.706+0000 7f8861ffb640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f885006b810 0x7f885006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:12.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.706+0000 7f8870d4a640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f885006b810 0x7f885006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:12.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.707+0000 7f8870d4a640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f885006b810 0x7f885006dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f886c15b320 tx=0x7f885c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:12.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.712+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f886807bac0 con 0x7f886c0cca30 2024-03-23T15:59:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.729+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8868077d60 con 0x7f886c0cca30 2024-03-23T15:59:12.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:12.951+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f886c06cf90 con 0x7f886c0cca30 2024-03-23T15:59:13.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.044+0000 7f8861ffb640 1 -- 172.21.15.129:0/3755068211 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f886807b420 con 0x7f886c0cca30 2024-03-23T15:59:13.047 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:13.047 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:13.047 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:13.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f885006b810 msgr2=0x7f885006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:13.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f885006b810 0x7f885006dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f886c15b320 tx=0x7f885c009040 comp rx=0 tx=0).stop 2024-03-23T15:59:13.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 msgr2=0x7f886c0c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:13.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 0x7f886c0c88a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f88680098f0 tx=0x7f8868004730 comp rx=0 tx=0).stop 2024-03-23T15:59:13.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 shutdown_connections 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f885006b810 0x7f885006dcd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f886c15df80 0x7f886c1497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f886c15dba0 0x7f886c1492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 --2- 172.21.15.129:0/3755068211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f886c0cca30 0x7f886c0c88a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 >> 172.21.15.129:0/3755068211 conn(0x7f886c071300 msgr2=0x7f886c0cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 shutdown_connections 2024-03-23T15:59:13.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:13.047+0000 7f88737af640 1 -- 172.21.15.129:0/3755068211 wait complete. 2024-03-23T15:59:13.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:13.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:14.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:14.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.767+0000 7f30e196c640 1 Processor -- start 2024-03-23T15:59:14.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.767+0000 7f30e196c640 1 -- start start 2024-03-23T15:59:14.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.768+0000 7f30e196c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f30dc072ad0 0x7f30dc06d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:14.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.768+0000 7f30e196c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f30dc0c7e70 0x7f30dc06def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:14.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.768+0000 7f30e196c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30dc073560 con 0x7f30dc150d00 2024-03-23T15:59:14.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.768+0000 7f30e196c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f30dc1510e0 con 0x7f30dc0c7e70 2024-03-23T15:59:14.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.768+0000 7f30e196c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f30dc072fc0 con 0x7f30dc072ad0 2024-03-23T15:59:14.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.768+0000 7f30db7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f30dc0c7e70 0x7f30dc06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:14.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.769+0000 7f30db7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f30dc0c7e70 0x7f30dc06def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45814/0 (socket says 172.21.15.129:45814) 2024-03-23T15:59:14.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.769+0000 7f30db7fe640 1 -- 172.21.15.129:0/536237418 learned_addr learned my addr 172.21.15.129:0/536237418 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:14.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.769+0000 7f30da7fc640 1 --2- 172.21.15.129:0/536237418 >> v2:172.21.15.157:3300/0 conn(0x7f30dc072ad0 0x7f30dc06d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:14.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.771+0000 7f30db7fe640 1 -- 172.21.15.129:0/536237418 >> v2:172.21.15.157:3300/0 conn(0x7f30dc072ad0 msgr2=0x7f30dc06d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:14.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.771+0000 7f30db7fe640 1 --2- 172.21.15.129:0/536237418 >> v2:172.21.15.157:3300/0 conn(0x7f30dc072ad0 0x7f30dc06d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:14.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.771+0000 7f30db7fe640 1 -- 172.21.15.129:0/536237418 >> v1:172.21.15.157:6789/0 conn(0x7f30dc150d00 legacy=0x7f30dc1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:14.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.771+0000 7f30db7fe640 1 -- 172.21.15.129:0/536237418 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30dc073280 con 0x7f30dc0c7e70 2024-03-23T15:59:14.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.773+0000 7f30db7fe640 1 --2- 172.21.15.129:0/536237418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc06def0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f30dc074b20 tx=0x7f30d402d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fe62e894294cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:14.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.777+0000 7f30d9ffb640 1 -- 172.21.15.129:0/536237418 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d402dc30 con 0x7f30dc0c7e70 2024-03-23T15:59:14.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.777+0000 7f30d9ffb640 1 -- 172.21.15.129:0/536237418 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d402dd90 con 0x7f30dc0c7e70 2024-03-23T15:59:14.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.777+0000 7f30d9ffb640 1 -- 172.21.15.129:0/536237418 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4035610 con 0x7f30dc0c7e70 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.777+0000 7f30e196c640 1 -- 172.21.15.129:0/536237418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 msgr2=0x7f30dc06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.777+0000 7f30e196c640 1 --2- 172.21.15.129:0/536237418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc06def0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f30dc074b20 tx=0x7f30d402d090 comp rx=0 tx=0).stop 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.778+0000 7f30e196c640 1 -- 172.21.15.129:0/536237418 shutdown_connections 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.778+0000 7f30e196c640 1 --2- 172.21.15.129:0/536237418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc06def0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.778+0000 7f30e196c640 1 --2- 172.21.15.129:0/536237418 >> v2:172.21.15.157:3300/0 conn(0x7f30dc072ad0 0x7f30dc06d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.778+0000 7f30e196c640 1 -- 172.21.15.129:0/536237418 >> 172.21.15.129:0/536237418 conn(0x7f30dc0c8370 msgr2=0x7f30dc0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.778+0000 7f30e196c640 1 -- 172.21.15.129:0/536237418 shutdown_connections 2024-03-23T15:59:14.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.778+0000 7f30e196c640 1 -- 172.21.15.129:0/536237418 wait complete. 2024-03-23T15:59:14.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.779+0000 7f30e196c640 1 Processor -- start 2024-03-23T15:59:14.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.779+0000 7f30e196c640 1 -- start start 2024-03-23T15:59:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.780+0000 7f30e196c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30dc072ad0 0x7f30dc1ec510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.780+0000 7f30e196c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc1e13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.780+0000 7f30e196c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30dc150d00 0x7f30dc0ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.780+0000 7f30e196c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc1eca50 con 0x7f30dc0c7e70 2024-03-23T15:59:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.780+0000 7f30e196c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc1dfb40 con 0x7f30dc150d00 2024-03-23T15:59:14.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.780+0000 7f30e196c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f30dc1dfde0 con 0x7f30dc072ad0 2024-03-23T15:59:14.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.781+0000 7f30da7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc1e13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:14.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.781+0000 7f30db7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30dc150d00 0x7f30dc0ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:14.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.781+0000 7f30daffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30dc072ad0 0x7f30dc1ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:14.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.781+0000 7f30da7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc1e13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45820/0 (socket says 172.21.15.129:45820) 2024-03-23T15:59:14.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.781+0000 7f30da7fc640 1 -- 172.21.15.129:0/907200602 learned_addr learned my addr 172.21.15.129:0/907200602 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:14.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.784+0000 7f30da7fc640 1 -- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30dc072ad0 msgr2=0x7f30dc1ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:14.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.784+0000 7f30da7fc640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30dc072ad0 0x7f30dc1ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:14.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.784+0000 7f30da7fc640 1 -- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30dc150d00 msgr2=0x7f30dc0ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:14.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.784+0000 7f30da7fc640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30dc150d00 0x7f30dc0ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:14.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.784+0000 7f30da7fc640 1 -- 172.21.15.129:0/907200602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d4009b30 con 0x7f30dc0c7e70 2024-03-23T15:59:14.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.786+0000 7f30da7fc640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc1e13f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f30d000e980 tx=0x7f30d000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:14.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.791+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0009670 con 0x7f30dc0c7e70 2024-03-23T15:59:14.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.791+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d000ca00 con 0x7f30dc0c7e70 2024-03-23T15:59:14.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.791+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30dc1e00c0 con 0x7f30dc0c7e70 2024-03-23T15:59:14.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.791+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d000cd90 con 0x7f30dc0c7e70 2024-03-23T15:59:14.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.791+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30dc0c9e40 con 0x7f30dc0c7e70 2024-03-23T15:59:14.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.792+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30dc075740 con 0x7f30dc0c7e70 2024-03-23T15:59:14.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.811+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30d0013470 con 0x7f30dc0c7e70 2024-03-23T15:59:14.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.811+0000 7f30c3fff640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30c806b860 0x7f30c806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:14.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.812+0000 7f30daffd640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30c806b860 0x7f30c806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:14.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:14.812+0000 7f30daffd640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30c806b860 0x7f30c806dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f30cc0016e0 tx=0x7f30cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:15.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.058+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f30d0014e40 con 0x7f30dc0c7e70 2024-03-23T15:59:15.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.058+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f30d0055600 con 0x7f30dc0c7e70 2024-03-23T15:59:15.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.284+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30dc156940 con 0x7f30dc0c7e70 2024-03-23T15:59:15.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.307+0000 7f30c3fff640 1 -- 172.21.15.129:0/907200602 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f30d0021090 con 0x7f30dc0c7e70 2024-03-23T15:59:15.310 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:15.310 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:15.310 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30c806b860 msgr2=0x7f30c806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30c806b860 0x7f30c806dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f30cc0016e0 tx=0x7f30cc009040 comp rx=0 tx=0).stop 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 msgr2=0x7f30dc1e13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc1e13f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f30d000e980 tx=0x7f30d000ee40 comp rx=0 tx=0).stop 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 shutdown_connections 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f30c806b860 0x7f30c806dd20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30dc150d00 0x7f30dc0ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:15.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0c7e70 0x7f30dc1e13f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:15.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 --2- 172.21.15.129:0/907200602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30dc072ad0 0x7f30dc1ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:15.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 >> 172.21.15.129:0/907200602 conn(0x7f30dc0c8370 msgr2=0x7f30dc14a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:15.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 shutdown_connections 2024-03-23T15:59:15.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:15.311+0000 7f30e196c640 1 -- 172.21.15.129:0/907200602 wait complete. 2024-03-23T15:59:15.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:15.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:16.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:17.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.033+0000 7f39f0f93640 1 Processor -- start 2024-03-23T15:59:17.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.033+0000 7f39f0f93640 1 -- start start 2024-03-23T15:59:17.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39f0f93640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f39ec06c030 0x7f39ec06c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:17.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39f0f93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:17.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39f0f93640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ec06cae0 con 0x7f39ec0cc080 2024-03-23T15:59:17.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39f0f93640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f39ec0cabc0 con 0x7f39ec06c030 2024-03-23T15:59:17.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39f0f93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec0caea0 con 0x7f39ec0cbc60 2024-03-23T15:59:17.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39ead9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f39ec0cc080 0x7f39ec15de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:47074/0 (socket says 172.21.15.129:47074) 2024-03-23T15:59:17.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.034+0000 7f39ead9d640 1 -- 172.21.15.129:0/1850109483 learned_addr learned my addr 172.21.15.129:0/1850109483 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:17.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.035+0000 7f39e9d9b640 1 --2- 172.21.15.129:0/1850109483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:17.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.035+0000 7f39ea59c640 1 --2- 172.21.15.129:0/1850109483 >> v2:172.21.15.157:3300/0 conn(0x7f39ec06c030 0x7f39ec06c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:17.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.037+0000 7f39e9d9b640 1 -- 172.21.15.129:0/1850109483 >> v2:172.21.15.157:3300/0 conn(0x7f39ec06c030 msgr2=0x7f39ec06c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:17.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.037+0000 7f39e9d9b640 1 --2- 172.21.15.129:0/1850109483 >> v2:172.21.15.157:3300/0 conn(0x7f39ec06c030 0x7f39ec06c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.037+0000 7f39e9d9b640 1 -- 172.21.15.129:0/1850109483 >> v1:172.21.15.129:6789/0 conn(0x7f39ec0cc080 legacy=0x7f39ec15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:17.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.037+0000 7f39e9d9b640 1 -- 172.21.15.129:0/1850109483 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec0cb180 con 0x7f39ec0cbc60 2024-03-23T15:59:17.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.040+0000 7f39e9d9b640 1 --2- 172.21.15.129:0/1850109483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec15d900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39e00097a0 tx=0x7f39e002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33cb63ca8cdc3076 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:17.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.044+0000 7f39e959a640 1 -- 172.21.15.129:0/1850109483 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e002e070 con 0x7f39ec0cbc60 2024-03-23T15:59:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.044+0000 7f39e959a640 1 -- 172.21.15.129:0/1850109483 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e00315a0 con 0x7f39ec0cbc60 2024-03-23T15:59:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.044+0000 7f39e959a640 1 -- 172.21.15.129:0/1850109483 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0031930 con 0x7f39ec0cbc60 2024-03-23T15:59:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 -- 172.21.15.129:0/1850109483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 msgr2=0x7f39ec15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 --2- 172.21.15.129:0/1850109483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec15d900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39e00097a0 tx=0x7f39e002b840 comp rx=0 tx=0).stop 2024-03-23T15:59:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 -- 172.21.15.129:0/1850109483 shutdown_connections 2024-03-23T15:59:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 --2- 172.21.15.129:0/1850109483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec15d900 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 --2- 172.21.15.129:0/1850109483 >> v2:172.21.15.157:3300/0 conn(0x7f39ec06c030 0x7f39ec06c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 -- 172.21.15.129:0/1850109483 >> 172.21.15.129:0/1850109483 conn(0x7f39ec0cfb20 msgr2=0x7f39ec0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 -- 172.21.15.129:0/1850109483 shutdown_connections 2024-03-23T15:59:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.045+0000 7f39f0f93640 1 -- 172.21.15.129:0/1850109483 wait complete. 2024-03-23T15:59:17.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 Processor -- start 2024-03-23T15:59:17.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 -- start start 2024-03-23T15:59:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39ec06c030 0x7f39ec14e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec1f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 0x7f39ec0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec15e7b0 con 0x7f39ec0cc080 2024-03-23T15:59:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec0c6680 con 0x7f39ec06c030 2024-03-23T15:59:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.047+0000 7f39f0f93640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec15df10 con 0x7f39ec0cbc60 2024-03-23T15:59:17.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.048+0000 7f39ead9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 0x7f39ec0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:17.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.048+0000 7f39ea59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39ec06c030 0x7f39ec14e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:17.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.048+0000 7f39ead9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 0x7f39ec0c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45832/0 (socket says 172.21.15.129:45832) 2024-03-23T15:59:17.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.048+0000 7f39ead9d640 1 -- 172.21.15.129:0/3089361372 learned_addr learned my addr 172.21.15.129:0/3089361372 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:17.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.048+0000 7f39e9d9b640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec1f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:17.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.052+0000 7f39ead9d640 1 -- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 msgr2=0x7f39ec1f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:17.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.052+0000 7f39ead9d640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec1f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.052+0000 7f39ead9d640 1 -- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39ec06c030 msgr2=0x7f39ec14e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:17.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.052+0000 7f39ead9d640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39ec06c030 0x7f39ec14e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.052+0000 7f39ead9d640 1 -- 172.21.15.129:0/3089361372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e0009400 con 0x7f39ec0cc080 2024-03-23T15:59:17.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.054+0000 7f39ead9d640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 0x7f39ec0c5680 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f39e400cbc0 tx=0x7f39e400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:17.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.061+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e400ad40 con 0x7f39ec0cc080 2024-03-23T15:59:17.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.061+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e4012400 con 0x7f39ec0cc080 2024-03-23T15:59:17.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.061+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e400aea0 con 0x7f39ec0cc080 2024-03-23T15:59:17.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.061+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec15e1f0 con 0x7f39ec0cc080 2024-03-23T15:59:17.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.061+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec0c4e80 con 0x7f39ec0cc080 2024-03-23T15:59:17.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.062+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ec0c4ce0 con 0x7f39ec0cc080 2024-03-23T15:59:17.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.341+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f39e4007070 con 0x7f39ec0cc080 2024-03-23T15:59:17.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.342+0000 7f39daffd640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f39cc06b810 0x7f39cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:17.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.342+0000 7f39ea59c640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f39cc06b810 0x7f39cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:17.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.342+0000 7f39ea59c640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f39cc06b810 0x7f39cc06dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009aa0 tx=0x7f39dc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:17.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.366+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f39e4058be0 con 0x7f39ec0cc080 2024-03-23T15:59:17.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.367+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f39e4054e80 con 0x7f39ec0cc080 2024-03-23T15:59:17.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.593+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39ec0cd900 con 0x7f39ec0cc080 2024-03-23T15:59:17.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.606+0000 7f39daffd640 1 -- 172.21.15.129:0/3089361372 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f39ec0cd900 con 0x7f39ec0cc080 2024-03-23T15:59:17.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:17.608 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:17.608 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f39cc06b810 msgr2=0x7f39cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f39cc06b810 0x7f39cc06dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009aa0 tx=0x7f39dc009190 comp rx=0 tx=0).stop 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 msgr2=0x7f39ec0c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 0x7f39ec0c5680 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f39e400cbc0 tx=0x7f39e400a430 comp rx=0 tx=0).stop 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 shutdown_connections 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39ec0cc080 0x7f39ec0c5680 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39ec0cbc60 0x7f39ec1f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f39cc06b810 0x7f39cc06dcd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 --2- 172.21.15.129:0/3089361372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39ec06c030 0x7f39ec14e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 >> 172.21.15.129:0/3089361372 conn(0x7f39ec0cfb20 msgr2=0x7f39ec0ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 shutdown_connections 2024-03-23T15:59:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:17.608+0000 7f39f0f93640 1 -- 172.21.15.129:0/3089361372 wait complete. 2024-03-23T15:59:18.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:18.227 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:19.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:19.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.329+0000 7fcc28d0c640 1 Processor -- start 2024-03-23T15:59:19.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.329+0000 7fcc28d0c640 1 -- start start 2024-03-23T15:59:19.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc28d0c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc2406c030 0x7fcc2406c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc28d0c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc240cbc60 0x7fcc2415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc28d0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc240cc080 0x7fcc2415de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc28d0c640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fcc2406cae0 con 0x7fcc2406c030 2024-03-23T15:59:19.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc28d0c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcc240cabc0 con 0x7fcc240cbc60 2024-03-23T15:59:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc28d0c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc240caea0 con 0x7fcc240cc080 2024-03-23T15:59:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc1bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc240cbc60 0x7fcc2415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc2259c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc2406c030 0x7fcc2406c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.330+0000 7fcc22d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc240cc080 0x7fcc2415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.331+0000 7fcc1bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc240cbc60 0x7fcc2415d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:33040/0 (socket says 172.21.15.129:33040) 2024-03-23T15:59:19.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.331+0000 7fcc1bfff640 1 -- 172.21.15.129:0/1868138672 learned_addr learned my addr 172.21.15.129:0/1868138672 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:19.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.334+0000 7fcc2259c640 1 -- 172.21.15.129:0/1868138672 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc240cc080 msgr2=0x7fcc2415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:19.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.334+0000 7fcc2259c640 1 --2- 172.21.15.129:0/1868138672 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc240cc080 0x7fcc2415de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.334+0000 7fcc2259c640 1 -- 172.21.15.129:0/1868138672 >> v2:172.21.15.157:3300/0 conn(0x7fcc240cbc60 msgr2=0x7fcc2415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:19.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.334+0000 7fcc2259c640 1 --2- 172.21.15.129:0/1868138672 >> v2:172.21.15.157:3300/0 conn(0x7fcc240cbc60 0x7fcc2415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.334+0000 7fcc2259c640 1 -- 172.21.15.129:0/1868138672 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc240cb180 con 0x7fcc2406c030 2024-03-23T15:59:19.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.337+0000 7fcc2259c640 1 --2- 172.21.15.129:0/1868138672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2406c030 0x7fcc2406c410 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcc240cd660 tx=0x7fcc1402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c44a05289e2570f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:19.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.341+0000 7fcc21d9b640 1 -- 172.21.15.129:0/1868138672 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1402e070 con 0x7fcc2406c030 2024-03-23T15:59:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.341+0000 7fcc21d9b640 1 -- 172.21.15.129:0/1868138672 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc140315a0 con 0x7fcc2406c030 2024-03-23T15:59:19.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.341+0000 7fcc21d9b640 1 -- 172.21.15.129:0/1868138672 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14031930 con 0x7fcc2406c030 2024-03-23T15:59:19.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1868138672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2406c030 msgr2=0x7fcc2406c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:19.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1868138672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2406c030 0x7fcc2406c410 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcc240cd660 tx=0x7fcc1402b840 comp rx=0 tx=0).stop 2024-03-23T15:59:19.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1868138672 shutdown_connections 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1868138672 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc240cc080 0x7fcc2415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1868138672 >> v2:172.21.15.157:3300/0 conn(0x7fcc240cbc60 0x7fcc2415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1868138672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2406c030 0x7fcc2406c410 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1868138672 >> 172.21.15.129:0/1868138672 conn(0x7fcc240cfb20 msgr2=0x7fcc240cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1868138672 shutdown_connections 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.342+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1868138672 wait complete. 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.343+0000 7fcc28d0c640 1 Processor -- start 2024-03-23T15:59:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 -- start start 2024-03-23T15:59:19.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc2406c030 0x7fcc2415db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 0x7fcc24158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc240cc080 0x7fcc24159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc240c3650 con 0x7fcc240cbc60 2024-03-23T15:59:19.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2415a660 con 0x7fcc240cc080 2024-03-23T15:59:19.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.344+0000 7fcc28d0c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc24158660 con 0x7fcc2406c030 2024-03-23T15:59:19.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.345+0000 7fcc1bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 0x7fcc24158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.345+0000 7fcc22d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc240cc080 0x7fcc24159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.345+0000 7fcc2259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc2406c030 0x7fcc2415db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.345+0000 7fcc1bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 0x7fcc24158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45856/0 (socket says 172.21.15.129:45856) 2024-03-23T15:59:19.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.345+0000 7fcc1bfff640 1 -- 172.21.15.129:0/1300319124 learned_addr learned my addr 172.21.15.129:0/1300319124 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.350+0000 7fcc1bfff640 1 -- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc2406c030 msgr2=0x7fcc2415db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.350+0000 7fcc1bfff640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc2406c030 0x7fcc2415db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.350+0000 7fcc1bfff640 1 -- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc240cc080 msgr2=0x7fcc24159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.350+0000 7fcc1bfff640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc240cc080 0x7fcc24159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.350+0000 7fcc1bfff640 1 -- 172.21.15.129:0/1300319124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc14009400 con 0x7fcc240cbc60 2024-03-23T15:59:19.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.353+0000 7fcc1bfff640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 0x7fcc24158af0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009fd0 tx=0x7fcc0c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.357+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00c8e0 con 0x7fcc240cbc60 2024-03-23T15:59:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.357+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00ca40 con 0x7fcc240cbc60 2024-03-23T15:59:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.357+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2415a220 con 0x7fcc240cbc60 2024-03-23T15:59:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.357+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00cdd0 con 0x7fcc240cbc60 2024-03-23T15:59:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.357+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2415b280 con 0x7fcc240cbc60 2024-03-23T15:59:19.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.358+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2415aa40 con 0x7fcc240cbc60 2024-03-23T15:59:19.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.611+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc0c013470 con 0x7fcc240cbc60 2024-03-23T15:59:19.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.612+0000 7fcc1affd640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0406b890 0x7fcc0406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:19.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.612+0000 7fcc2259c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0406b890 0x7fcc0406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:19.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.613+0000 7fcc2259c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0406b890 0x7fcc0406dd50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcc14036000 tx=0x7fcc1403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:19.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.627+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcc0c014e40 con 0x7fcc240cbc60 2024-03-23T15:59:19.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.627+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc0c055500 con 0x7fcc240cbc60 2024-03-23T15:59:19.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.852+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc24075740 con 0x7fcc240cbc60 2024-03-23T15:59:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.872+0000 7fcc1affd640 1 -- 172.21.15.129:0/1300319124 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcc0c021090 con 0x7fcc240cbc60 2024-03-23T15:59:19.875 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:19.875 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.874+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0406b890 msgr2=0x7fcc0406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0406b890 0x7fcc0406dd50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcc14036000 tx=0x7fcc1403d040 comp rx=0 tx=0).stop 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 msgr2=0x7fcc24158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 0x7fcc24158af0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009fd0 tx=0x7fcc0c00ed40 comp rx=0 tx=0).stop 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 shutdown_connections 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0406b890 0x7fcc0406dd50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc240cc080 0x7fcc24159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cbc60 0x7fcc24158af0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 --2- 172.21.15.129:0/1300319124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc2406c030 0x7fcc2415db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 >> 172.21.15.129:0/1300319124 conn(0x7fcc240cfb20 msgr2=0x7fcc240ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 shutdown_connections 2024-03-23T15:59:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:19.875+0000 7fcc28d0c640 1 -- 172.21.15.129:0/1300319124 wait complete. 2024-03-23T15:59:20.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:20.492 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:21.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:21.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.593+0000 7f0764f6b640 1 Processor -- start 2024-03-23T15:59:21.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.593+0000 7f0764f6b640 1 -- start start 2024-03-23T15:59:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.594+0000 7f0764f6b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ccd60 0x7f076015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.594+0000 7f0764f6b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f07600ceb00 0x7f076006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.594+0000 7f0764f6b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f07600cef40 0x7f076006c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.594+0000 7f0764f6b640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f076015b290 con 0x7f07600ceb00 2024-03-23T15:59:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.594+0000 7f0764f6b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f07600ce800 con 0x7f07600cef40 2024-03-23T15:59:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.594+0000 7f0764f6b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0760154a70 con 0x7f07600ccd60 2024-03-23T15:59:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.595+0000 7f075e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ccd60 0x7f076015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.595+0000 7f075ed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f07600cef40 0x7f076006c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.595+0000 7f0757fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f07600ceb00 0x7f076006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.596+0000 7f075e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ccd60 0x7f076015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:39996/0 (socket says 172.21.15.129:39996) 2024-03-23T15:59:21.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.596+0000 7f075e59c640 1 -- 172.21.15.129:0/3502896072 learned_addr learned my addr 172.21.15.129:0/3502896072 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.598+0000 7f0757fff640 1 -- 172.21.15.129:0/3502896072 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ccd60 msgr2=0x7f076015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.598+0000 7f0757fff640 1 --2- 172.21.15.129:0/3502896072 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ccd60 0x7f076015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.598+0000 7f0757fff640 1 -- 172.21.15.129:0/3502896072 >> v2:172.21.15.157:3300/0 conn(0x7f07600cef40 msgr2=0x7f076006c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.598+0000 7f0757fff640 1 --2- 172.21.15.129:0/3502896072 >> v2:172.21.15.157:3300/0 conn(0x7f07600cef40 0x7f076006c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.598+0000 7f0757fff640 1 -- 172.21.15.129:0/3502896072 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07600cd950 con 0x7f07600ceb00 2024-03-23T15:59:21.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.600+0000 7f0757fff640 1 --2- 172.21.15.129:0/3502896072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ceb00 0x7f076006c270 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f076015e8e0 tx=0x7f074802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa281de4d70d51b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:21.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.603+0000 7f075dd9b640 1 -- 172.21.15.129:0/3502896072 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748031830 con 0x7f07600ceb00 2024-03-23T15:59:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.605+0000 7f075dd9b640 1 -- 172.21.15.129:0/3502896072 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0748031990 con 0x7f07600ceb00 2024-03-23T15:59:21.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.605+0000 7f075dd9b640 1 -- 172.21.15.129:0/3502896072 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748031d20 con 0x7f07600ceb00 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 -- 172.21.15.129:0/3502896072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ceb00 msgr2=0x7f076006c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3502896072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ceb00 0x7f076006c270 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f076015e8e0 tx=0x7f074802b8d0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 -- 172.21.15.129:0/3502896072 shutdown_connections 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3502896072 >> v2:172.21.15.157:3300/0 conn(0x7f07600cef40 0x7f076006c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3502896072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ceb00 0x7f076006c270 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3502896072 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ccd60 0x7f076015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 -- 172.21.15.129:0/3502896072 >> 172.21.15.129:0/3502896072 conn(0x7f07600b1840 msgr2=0x7f07600b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 -- 172.21.15.129:0/3502896072 shutdown_connections 2024-03-23T15:59:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.606+0000 7f0764f6b640 1 -- 172.21.15.129:0/3502896072 wait complete. 2024-03-23T15:59:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.607+0000 7f0764f6b640 1 Processor -- start 2024-03-23T15:59:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.607+0000 7f0764f6b640 1 -- start start 2024-03-23T15:59:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0764f6b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 0x7f07600c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0764f6b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ceb00 0x7f07600b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0764f6b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07600cef40 0x7f076015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0764f6b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07601f1730 con 0x7f07600ccd60 2024-03-23T15:59:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0764f6b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07601eaef0 con 0x7f07600cef40 2024-03-23T15:59:21.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0764f6b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f07601e3da0 con 0x7f07600ceb00 2024-03-23T15:59:21.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f075e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 0x7f07600c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.608+0000 7f0757fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ceb00 0x7f07600b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.609+0000 7f075ed9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07600cef40 0x7f076015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.609+0000 7f075e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 0x7f07600c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52998/0 (socket says 172.21.15.129:52998) 2024-03-23T15:59:21.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.609+0000 7f075e59c640 1 -- 172.21.15.129:0/3790730578 learned_addr learned my addr 172.21.15.129:0/3790730578 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:21.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.612+0000 7f075e59c640 1 -- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ceb00 msgr2=0x7f07600b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:21.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.612+0000 7f075e59c640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ceb00 0x7f07600b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.612+0000 7f075e59c640 1 -- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07600cef40 msgr2=0x7f076015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:21.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.612+0000 7f075e59c640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07600cef40 0x7f076015dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:21.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.612+0000 7f075e59c640 1 -- 172.21.15.129:0/3790730578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0748009400 con 0x7f07600ccd60 2024-03-23T15:59:21.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.614+0000 7f075e59c640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 0x7f07600c9440 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f075000e870 tx=0x7f075000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:21.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.619+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075000c8e0 con 0x7f07600ccd60 2024-03-23T15:59:21.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.619+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f075000ca40 con 0x7f07600ccd60 2024-03-23T15:59:21.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.619+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07601f1450 con 0x7f07600ccd60 2024-03-23T15:59:21.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.619+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075000cdf0 con 0x7f07600ccd60 2024-03-23T15:59:21.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.619+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07601e3ac0 con 0x7f07600ccd60 2024-03-23T15:59:21.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.620+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f076004f530 con 0x7f07600ccd60 2024-03-23T15:59:21.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.642+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0750013470 con 0x7f07600ccd60 2024-03-23T15:59:21.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.643+0000 7f0756ffd640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f074006b810 0x7f074006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:21.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.643+0000 7f0757fff640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f074006b810 0x7f074006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:21.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.644+0000 7f0757fff640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f074006b810 0x7f074006dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0748036000 tx=0x7f074803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.898+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0750014e40 con 0x7f07600ccd60 2024-03-23T15:59:21.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:21.898+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0750055630 con 0x7f07600ccd60 2024-03-23T15:59:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.116+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f076006db50 con 0x7f07600ccd60 2024-03-23T15:59:22.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.129+0000 7f0756ffd640 1 -- 172.21.15.129:0/3790730578 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0750017090 con 0x7f07600ccd60 2024-03-23T15:59:22.131 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:22.132 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:22.132 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:22.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f074006b810 msgr2=0x7f074006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:22.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f074006b810 0x7f074006dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0748036000 tx=0x7f074803c040 comp rx=0 tx=0).stop 2024-03-23T15:59:22.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 msgr2=0x7f07600c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:22.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 0x7f07600c9440 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f075000e870 tx=0x7f075000ed30 comp rx=0 tx=0).stop 2024-03-23T15:59:22.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 shutdown_connections 2024-03-23T15:59:22.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f074006b810 0x7f074006dcd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:22.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07600cef40 0x7f076015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:22.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07600ceb00 0x7f07600b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:22.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 --2- 172.21.15.129:0/3790730578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07600ccd60 0x7f07600c9440 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:22.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 >> 172.21.15.129:0/3790730578 conn(0x7f07600b1840 msgr2=0x7f07600b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:22.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 shutdown_connections 2024-03-23T15:59:22.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:22.132+0000 7f0764f6b640 1 -- 172.21.15.129:0/3790730578 wait complete. 2024-03-23T15:59:22.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:22.749 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:23.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:23.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.851+0000 7f55b0fc2640 1 Processor -- start 2024-03-23T15:59:23.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.851+0000 7f55b0fc2640 1 -- start start 2024-03-23T15:59:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.852+0000 7f55b0fc2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f55ac15dc60 0x7f55ac06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.852+0000 7f55b0fc2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f55ac06f570 0x7f55ac0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.852+0000 7f55b0fc2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ac06d2c0 con 0x7f55ac15e040 2024-03-23T15:59:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.852+0000 7f55b0fc2640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f55ac15e740 con 0x7f55ac06f570 2024-03-23T15:59:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.852+0000 7f55b0fc2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f55ac0ccd00 con 0x7f55ac15dc60 2024-03-23T15:59:23.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.853+0000 7f55aad9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f55ac06f570 0x7f55ac0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:23.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.853+0000 7f55aad9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f55ac06f570 0x7f55ac0cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53012/0 (socket says 172.21.15.129:53012) 2024-03-23T15:59:23.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.853+0000 7f55aad9d640 1 -- 172.21.15.129:0/348044696 learned_addr learned my addr 172.21.15.129:0/348044696 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:23.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.853+0000 7f55aa59c640 1 --2- 172.21.15.129:0/348044696 >> v2:172.21.15.157:3300/0 conn(0x7f55ac15dc60 0x7f55ac06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:23.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.855+0000 7f55aad9d640 1 -- 172.21.15.129:0/348044696 >> v2:172.21.15.157:3300/0 conn(0x7f55ac15dc60 msgr2=0x7f55ac06cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:23.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.855+0000 7f55aad9d640 1 --2- 172.21.15.129:0/348044696 >> v2:172.21.15.157:3300/0 conn(0x7f55ac15dc60 0x7f55ac06cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:23.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.855+0000 7f55aad9d640 1 -- 172.21.15.129:0/348044696 >> v1:172.21.15.157:6789/0 conn(0x7f55ac15e040 legacy=0x7f55ac0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:23.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.855+0000 7f55aad9d640 1 -- 172.21.15.129:0/348044696 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ac0ccfe0 con 0x7f55ac06f570 2024-03-23T15:59:23.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.857+0000 7f55aad9d640 1 --2- 172.21.15.129:0/348044696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac06f570 0x7f55ac0cb7c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0732d0 tx=0x7f55a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d7a72856e2d0df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:23.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.861+0000 7f55a959a640 1 -- 172.21.15.129:0/348044696 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a402ca80 con 0x7f55ac06f570 2024-03-23T15:59:23.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.862+0000 7f55a959a640 1 -- 172.21.15.129:0/348044696 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a402cbe0 con 0x7f55ac06f570 2024-03-23T15:59:23.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.862+0000 7f55a959a640 1 -- 172.21.15.129:0/348044696 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a40343f0 con 0x7f55ac06f570 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 -- 172.21.15.129:0/348044696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac06f570 msgr2=0x7f55ac0cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/348044696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac06f570 0x7f55ac0cb7c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0732d0 tx=0x7f55a402bee0 comp rx=0 tx=0).stop 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 -- 172.21.15.129:0/348044696 shutdown_connections 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/348044696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac06f570 0x7f55ac0cb7c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/348044696 >> v2:172.21.15.157:3300/0 conn(0x7f55ac15dc60 0x7f55ac06cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 -- 172.21.15.129:0/348044696 >> 172.21.15.129:0/348044696 conn(0x7f55ac071300 msgr2=0x7f55ac0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 -- 172.21.15.129:0/348044696 shutdown_connections 2024-03-23T15:59:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.863+0000 7f55b0fc2640 1 -- 172.21.15.129:0/348044696 wait complete. 2024-03-23T15:59:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.865+0000 7f55b0fc2640 1 Processor -- start 2024-03-23T15:59:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.865+0000 7f55b0fc2640 1 -- start start 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55b0fc2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ac06f570 0x7f55ac0c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55b0fc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 0x7f55ac149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55b0fc2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ac15e040 0x7f55ac149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55b0fc2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac14c4d0 con 0x7f55ac15dc60 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55b0fc2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac14bb70 con 0x7f55ac06f570 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55b0fc2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55ac0c7780 con 0x7f55ac15e040 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.866+0000 7f55a9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 0x7f55ac149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.867+0000 7f55aa59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ac06f570 0x7f55ac0c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.867+0000 7f55a9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 0x7f55ac149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53024/0 (socket says 172.21.15.129:53024) 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.867+0000 7f55a9d9b640 1 -- 172.21.15.129:0/3991967015 learned_addr learned my addr 172.21.15.129:0/3991967015 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.867+0000 7f55aad9d640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ac15e040 0x7f55ac149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:23.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.870+0000 7f55a9d9b640 1 -- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ac15e040 msgr2=0x7f55ac149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:23.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.870+0000 7f55a9d9b640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ac15e040 0x7f55ac149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:23.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.870+0000 7f55a9d9b640 1 -- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ac06f570 msgr2=0x7f55ac0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:23.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.870+0000 7f55a9d9b640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ac06f570 0x7f55ac0c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:23.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.870+0000 7f55a9d9b640 1 -- 172.21.15.129:0/3991967015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a4009b30 con 0x7f55ac15dc60 2024-03-23T15:59:23.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.872+0000 7f55a9d9b640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 0x7f55ac149310 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f55a000c440 tx=0x7f55a0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:23.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.876+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a00052d0 con 0x7f55ac15dc60 2024-03-23T15:59:23.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.876+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a0005430 con 0x7f55ac15dc60 2024-03-23T15:59:23.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.876+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0007260 con 0x7f55ac15dc60 2024-03-23T15:59:23.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.876+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ac14be10 con 0x7f55ac15dc60 2024-03-23T15:59:23.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.876+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ac0c7a60 con 0x7f55ac15dc60 2024-03-23T15:59:23.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:23.877+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ac14c610 con 0x7f55ac15dc60 2024-03-23T15:59:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.142+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f55a00073c0 con 0x7f55ac15dc60 2024-03-23T15:59:24.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.143+0000 7f559affd640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f558c06b890 0x7f558c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:24.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.143+0000 7f55aa59c640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f558c06b890 0x7f558c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:24.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.143+0000 7f55aa59c640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f558c06b890 0x7f558c06dd50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f559c0079a0 tx=0x7f559c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.164+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f55a00580b0 con 0x7f55ac15dc60 2024-03-23T15:59:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.164+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f55a0054350 con 0x7f55ac15dc60 2024-03-23T15:59:24.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.382+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55ac14c610 con 0x7f55ac15dc60 2024-03-23T15:59:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.402+0000 7f559affd640 1 -- 172.21.15.129:0/3991967015 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f55ac14c610 con 0x7f55ac15dc60 2024-03-23T15:59:24.404 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:24.404 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:24.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f558c06b890 msgr2=0x7f558c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f558c06b890 0x7f558c06dd50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f559c0079a0 tx=0x7f559c008040 comp rx=0 tx=0).stop 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 msgr2=0x7f55ac149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 0x7f55ac149310 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f55a000c440 tx=0x7f55a0001e70 comp rx=0 tx=0).stop 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 shutdown_connections 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ac15e040 0x7f55ac149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ac15dc60 0x7f55ac149310 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f558c06b890 0x7f558c06dd50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 --2- 172.21.15.129:0/3991967015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ac06f570 0x7f55ac0c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 >> 172.21.15.129:0/3991967015 conn(0x7f55ac071300 msgr2=0x7f55ac0c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 shutdown_connections 2024-03-23T15:59:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:24.405+0000 7f55b0fc2640 1 -- 172.21.15.129:0/3991967015 wait complete. 2024-03-23T15:59:25.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:25.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:26.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:26.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.125+0000 7fea3a0fb640 1 Processor -- start 2024-03-23T15:59:26.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.125+0000 7fea3a0fb640 1 -- start start 2024-03-23T15:59:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.126+0000 7fea3a0fb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fea341544b0 0x7fea3415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.126+0000 7fea3a0fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea340cbcb0 con 0x7fea340caa00 2024-03-23T15:59:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.126+0000 7fea3a0fb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea3415b280 con 0x7fea340c6000 2024-03-23T15:59:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.126+0000 7fea3a0fb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fea340c5be0 con 0x7fea341544b0 2024-03-23T15:59:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.127+0000 7fea32ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fea340caa00 0x7fea3406bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57222/0 (socket says 172.21.15.129:57222) 2024-03-23T15:59:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.127+0000 7fea32ffd640 1 -- 172.21.15.129:0/2586177553 learned_addr learned my addr 172.21.15.129:0/2586177553 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.127+0000 7fea337fe640 1 --2- 172.21.15.129:0/2586177553 >> v2:172.21.15.129:3300/0 conn(0x7fea341544b0 0x7fea3415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:26.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.129+0000 7fea327fc640 1 -- 172.21.15.129:0/2586177553 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898245554 0 0) 0x7fea3415b280 con 0x7fea340c6000 2024-03-23T15:59:26.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.129+0000 7fea327fc640 1 -- 172.21.15.129:0/2586177553 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea1c003520 con 0x7fea340c6000 2024-03-23T15:59:26.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.132+0000 7fea337fe640 1 -- 172.21.15.129:0/2586177553 >> v1:172.21.15.157:6789/0 conn(0x7fea340c6000 legacy=0x7fea34164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.132+0000 7fea337fe640 1 -- 172.21.15.129:0/2586177553 >> v1:172.21.15.129:6789/0 conn(0x7fea340caa00 legacy=0x7fea3406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.132+0000 7fea337fe640 1 -- 172.21.15.129:0/2586177553 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3415a9e0 con 0x7fea341544b0 2024-03-23T15:59:26.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.135+0000 7fea337fe640 1 --2- 172.21.15.129:0/2586177553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea341544b0 0x7fea3415a2e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fea240097a0 tx=0x7fea2402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c0afa2ad93871c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:26.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea327fc640 1 -- 172.21.15.129:0/2586177553 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2402f810 con 0x7fea341544b0 2024-03-23T15:59:26.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea327fc640 1 -- 172.21.15.129:0/2586177553 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea2402f970 con 0x7fea341544b0 2024-03-23T15:59:26.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea327fc640 1 -- 172.21.15.129:0/2586177553 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2402fd00 con 0x7fea341544b0 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea3a0fb640 1 -- 172.21.15.129:0/2586177553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea341544b0 msgr2=0x7fea3415a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/2586177553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea341544b0 0x7fea3415a2e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fea240097a0 tx=0x7fea2402b9f0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea3a0fb640 1 -- 172.21.15.129:0/2586177553 shutdown_connections 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/2586177553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea341544b0 0x7fea3415a2e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea3a0fb640 1 -- 172.21.15.129:0/2586177553 >> 172.21.15.129:0/2586177553 conn(0x7fea34061b50 msgr2=0x7fea34061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.140+0000 7fea3a0fb640 1 -- 172.21.15.129:0/2586177553 shutdown_connections 2024-03-23T15:59:26.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.141+0000 7fea3a0fb640 1 -- 172.21.15.129:0/2586177553 wait complete. 2024-03-23T15:59:26.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.142+0000 7fea3a0fb640 1 Processor -- start 2024-03-23T15:59:26.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.142+0000 7fea3a0fb640 1 -- start start 2024-03-23T15:59:26.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea3a0fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 0x7fea340d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:26.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea3a0fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea340caa00 0x7fea34145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:26.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea3a0fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea341544b0 0x7fea34146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea3a0fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fea341f0e70 con 0x7fea340c6000 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea3a0fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea341f0310 con 0x7fea341544b0 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea3a0fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fea341f05d0 con 0x7fea340caa00 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 0x7fea340d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea33fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea341544b0 0x7fea34146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.143+0000 7fea32ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea340caa00 0x7fea34145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.144+0000 7fea337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 0x7fea340d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53042/0 (socket says 172.21.15.129:53042) 2024-03-23T15:59:26.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.144+0000 7fea337fe640 1 -- 172.21.15.129:0/3553002438 learned_addr learned my addr 172.21.15.129:0/3553002438 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:26.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.148+0000 7fea337fe640 1 -- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea340caa00 msgr2=0x7fea34145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.148+0000 7fea337fe640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea340caa00 0x7fea34145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.148+0000 7fea337fe640 1 -- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea341544b0 msgr2=0x7fea34146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.148+0000 7fea337fe640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea341544b0 0x7fea34146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.148+0000 7fea337fe640 1 -- 172.21.15.129:0/3553002438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea24009400 con 0x7fea340c6000 2024-03-23T15:59:26.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.152+0000 7fea337fe640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 0x7fea340d0180 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fea2402c000 tx=0x7fea24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.157+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2403c040 con 0x7fea340c6000 2024-03-23T15:59:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.157+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea240378f0 con 0x7fea340c6000 2024-03-23T15:59:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.157+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea341f08b0 con 0x7fea340c6000 2024-03-23T15:59:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.157+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24037ca0 con 0x7fea340c6000 2024-03-23T15:59:26.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.157+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea34145aa0 con 0x7fea340c6000 2024-03-23T15:59:26.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.158+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea3414e530 con 0x7fea340c6000 2024-03-23T15:59:26.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.187+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea2404b3b0 con 0x7fea340c6000 2024-03-23T15:59:26.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.188+0000 7fea1bfff640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea2006b810 0x7fea2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:26.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.188+0000 7fea32ffd640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea2006b810 0x7fea2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:26.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.189+0000 7fea32ffd640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea2006b810 0x7fea2006dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fea341edb20 tx=0x7fea28009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:26.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.192+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fea2407ad00 con 0x7fea340c6000 2024-03-23T15:59:26.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.445+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea24076fa0 con 0x7fea340c6000 2024-03-23T15:59:26.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.695+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea340b3450 con 0x7fea340c6000 2024-03-23T15:59:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.739+0000 7fea1bfff640 1 -- 172.21.15.129:0/3553002438 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fea2407a660 con 0x7fea340c6000 2024-03-23T15:59:26.743 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:26.743 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.741+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea2006b810 msgr2=0x7fea2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.741+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea2006b810 0x7fea2006dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fea341edb20 tx=0x7fea28009040 comp rx=0 tx=0).stop 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.741+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 msgr2=0x7fea340d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.741+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 0x7fea340d0180 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fea2402c000 tx=0x7fea24004730 comp rx=0 tx=0).stop 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 shutdown_connections 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fea2006b810 0x7fea2006dcd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea341544b0 0x7fea34146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea340caa00 0x7fea34145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 --2- 172.21.15.129:0/3553002438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea340c6000 0x7fea340d0180 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 >> 172.21.15.129:0/3553002438 conn(0x7fea34061b50 msgr2=0x7fea340c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 shutdown_connections 2024-03-23T15:59:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:26.742+0000 7fea3a0fb640 1 -- 172.21.15.129:0/3553002438 wait complete. 2024-03-23T15:59:27.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:27.362 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:28.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:28.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.463+0000 7fa05b713640 1 Processor -- start 2024-03-23T15:59:28.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.463+0000 7fa05b713640 1 -- start start 2024-03-23T15:59:28.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.464+0000 7fa05b713640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa054154270 0x7fa0540ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:28.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.464+0000 7fa05b713640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa05415ad10 0x7fa05406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:28.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.464+0000 7fa05b713640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa05415d880 con 0x7fa05406c650 2024-03-23T15:59:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.464+0000 7fa05b713640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa0540c4ae0 con 0x7fa054154270 2024-03-23T15:59:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.464+0000 7fa05b713640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa0540cd5e0 con 0x7fa05415ad10 2024-03-23T15:59:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.465+0000 7fa0594af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa054154270 0x7fa0540ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.465+0000 7fa058cae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa05415ad10 0x7fa05406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.465+0000 7fa059cb0640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa05406c650 0x7fa0540c4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:52262/0 (socket says 172.21.15.129:52262) 2024-03-23T15:59:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.465+0000 7fa059cb0640 1 -- 172.21.15.129:0/3298032490 learned_addr learned my addr 172.21.15.129:0/3298032490 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.468+0000 7fa0594af640 1 -- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa05415ad10 msgr2=0x7fa05406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:28.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.468+0000 7fa0594af640 1 --2- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa05415ad10 0x7fa05406c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.468+0000 7fa0594af640 1 -- 172.21.15.129:0/3298032490 >> v1:172.21.15.157:6789/0 conn(0x7fa05406c650 legacy=0x7fa0540c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.469+0000 7fa0594af640 1 -- 172.21.15.129:0/3298032490 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05415dab0 con 0x7fa054154270 2024-03-23T15:59:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.474+0000 7fa0594af640 1 --2- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa054154270 0x7fa0540ca580 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa0500097a0 tx=0x7fa05002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e0e795f784d515d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.478+0000 7fa04bfff640 1 -- 172.21.15.129:0/3298032490 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05002f050 con 0x7fa054154270 2024-03-23T15:59:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.478+0000 7fa04bfff640 1 -- 172.21.15.129:0/3298032490 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa050031700 con 0x7fa054154270 2024-03-23T15:59:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.478+0000 7fa04bfff640 1 -- 172.21.15.129:0/3298032490 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050031a90 con 0x7fa054154270 2024-03-23T15:59:28.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 -- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa054154270 msgr2=0x7fa0540ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:28.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 --2- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa054154270 0x7fa0540ca580 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa0500097a0 tx=0x7fa05002b7a0 comp rx=0 tx=0).stop 2024-03-23T15:59:28.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 -- 172.21.15.129:0/3298032490 shutdown_connections 2024-03-23T15:59:28.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 --2- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa05415ad10 0x7fa05406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:28.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 --2- 172.21.15.129:0/3298032490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa054154270 0x7fa0540ca580 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:28.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 -- 172.21.15.129:0/3298032490 >> 172.21.15.129:0/3298032490 conn(0x7fa05406d290 msgr2=0x7fa054072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:28.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 -- 172.21.15.129:0/3298032490 shutdown_connections 2024-03-23T15:59:28.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.479+0000 7fa05b713640 1 -- 172.21.15.129:0/3298032490 wait complete. 2024-03-23T15:59:28.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.481+0000 7fa05b713640 1 Processor -- start 2024-03-23T15:59:28.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.481+0000 7fa05b713640 1 -- start start 2024-03-23T15:59:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.482+0000 7fa05b713640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05406c650 0x7fa0540ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:28.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.482+0000 7fa05b713640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa054154270 0x7fa0540cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.482+0000 7fa05b713640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa05415ad10 0x7fa054158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.482+0000 7fa05b713640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa05415d1c0 con 0x7fa05415ad10 2024-03-23T15:59:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.482+0000 7fa05b713640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa0540ce9e0 con 0x7fa05406c650 2024-03-23T15:59:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.482+0000 7fa05b713640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa0541583a0 con 0x7fa054154270 2024-03-23T15:59:28.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.483+0000 7fa059cb0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa05415ad10 0x7fa054158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.483+0000 7fa058cae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa054154270 0x7fa0540cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.483+0000 7fa0594af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05406c650 0x7fa0540ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.483+0000 7fa059cb0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa05415ad10 0x7fa054158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53048/0 (socket says 172.21.15.129:53048) 2024-03-23T15:59:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.483+0000 7fa059cb0640 1 -- 172.21.15.129:0/4089837847 learned_addr learned my addr 172.21.15.129:0/4089837847 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.487+0000 7fa0594af640 1 -- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa054154270 msgr2=0x7fa0540cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.487+0000 7fa0594af640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa054154270 0x7fa0540cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.487+0000 7fa0594af640 1 -- 172.21.15.129:0/4089837847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa05415ad10 msgr2=0x7fa054158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.487+0000 7fa0594af640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa05415ad10 0x7fa054158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.487+0000 7fa0594af640 1 -- 172.21.15.129:0/4089837847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa050009400 con 0x7fa05406c650 2024-03-23T15:59:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.489+0000 7fa0594af640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05406c650 0x7fa0540ce4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa050033000 tx=0x7fa0500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.492+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050033bb0 con 0x7fa05406c650 2024-03-23T15:59:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.492+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05415cf20 con 0x7fa05406c650 2024-03-23T15:59:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.492+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0540cf550 con 0x7fa05406c650 2024-03-23T15:59:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.493+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa054158580 con 0x7fa05406c650 2024-03-23T15:59:28.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.703+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa050033d10 con 0x7fa05406c650 2024-03-23T15:59:28.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.790+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05002f020 con 0x7fa05406c650 2024-03-23T15:59:28.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.791+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa05003a070 con 0x7fa05406c650 2024-03-23T15:59:28.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.792+0000 7fa049ffb640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa03806b840 0x7fa03806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:28.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.792+0000 7fa058cae640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa03806b840 0x7fa03806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.793+0000 7fa058cae640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa03806b840 0x7fa03806dd00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa05415bc00 tx=0x7fa044009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.798+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa05007a070 con 0x7fa05406c650 2024-03-23T15:59:28.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:28.814+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa050076310 con 0x7fa05406c650 2024-03-23T15:59:29.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.078+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa054158580 con 0x7fa05406c650 2024-03-23T15:59:29.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.239+0000 7fa049ffb640 1 -- 172.21.15.129:0/4089837847 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa054158580 con 0x7fa05406c650 2024-03-23T15:59:29.242 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:29.242 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:29.242 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa03806b840 msgr2=0x7fa03806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa03806b840 0x7fa03806dd00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa05415bc00 tx=0x7fa044009190 comp rx=0 tx=0).stop 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05406c650 msgr2=0x7fa0540ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05406c650 0x7fa0540ce4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa050033000 tx=0x7fa0500046c0 comp rx=0 tx=0).stop 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 shutdown_connections 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa05415ad10 0x7fa054158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa054154270 0x7fa0540cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa03806b840 0x7fa03806dd00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:29.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 --2- 172.21.15.129:0/4089837847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa05406c650 0x7fa0540ce4a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:29.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 >> 172.21.15.129:0/4089837847 conn(0x7fa05406d290 msgr2=0x7fa05414f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:29.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 shutdown_connections 2024-03-23T15:59:29.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:29.242+0000 7fa05b713640 1 -- 172.21.15.129:0/4089837847 wait complete. 2024-03-23T15:59:29.274 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:49.117 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.327 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:49.170 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.372 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:49.171 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.377 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:49.175 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.381 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:12:49.180 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.384 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:12:49.183 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.399 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:12:49.242 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.456 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:12:49.255 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.500 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:12:49.299 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T15:59:29.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:29.861 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:30.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:30.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.961+0000 7fe147423640 1 Processor -- start 2024-03-23T15:59:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.962+0000 7fe147423640 1 -- start start 2024-03-23T15:59:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.963+0000 7fe147423640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe14006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.963+0000 7fe147423640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe1400cc080 0x7fe14015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.963+0000 7fe147423640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe14006cae0 con 0x7fe1400cbc60 2024-03-23T15:59:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.963+0000 7fe147423640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe1400cabc0 con 0x7fe1400cc080 2024-03-23T15:59:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.963+0000 7fe147423640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe1400caea0 con 0x7fe14006c030 2024-03-23T15:59:30.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.964+0000 7fe1459c0640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe1400cc080 0x7fe14015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:30.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.964+0000 7fe1449be640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe1400cbc60 0x7fe14015d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:49416/0 (socket says 172.21.15.129:49416) 2024-03-23T15:59:30.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.964+0000 7fe1449be640 1 -- 172.21.15.129:0/3232661803 learned_addr learned my addr 172.21.15.129:0/3232661803 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:30.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.964+0000 7fe1451bf640 1 --2- 172.21.15.129:0/3232661803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe14006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.966+0000 7fe1459c0640 1 -- 172.21.15.129:0/3232661803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 msgr2=0x7fe14006c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.966+0000 7fe1459c0640 1 --2- 172.21.15.129:0/3232661803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe14006c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.966+0000 7fe1459c0640 1 -- 172.21.15.129:0/3232661803 >> v1:172.21.15.157:6789/0 conn(0x7fe1400cbc60 legacy=0x7fe14015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.966+0000 7fe1459c0640 1 -- 172.21.15.129:0/3232661803 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1400cb180 con 0x7fe1400cc080 2024-03-23T15:59:30.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.968+0000 7fe1459c0640 1 --2- 172.21.15.129:0/3232661803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cc080 0x7fe14015e010 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe138009d90 tx=0x7fe13802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ecf2a61aa9430a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:30.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe137fff640 1 -- 172.21.15.129:0/3232661803 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13802cbc0 con 0x7fe1400cc080 2024-03-23T15:59:30.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe137fff640 1 -- 172.21.15.129:0/3232661803 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13802cd20 con 0x7fe1400cc080 2024-03-23T15:59:30.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe137fff640 1 -- 172.21.15.129:0/3232661803 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe138034660 con 0x7fe1400cc080 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe147423640 1 -- 172.21.15.129:0/3232661803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cc080 msgr2=0x7fe14015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe147423640 1 --2- 172.21.15.129:0/3232661803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cc080 0x7fe14015e010 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe138009d90 tx=0x7fe13802be10 comp rx=0 tx=0).stop 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe147423640 1 -- 172.21.15.129:0/3232661803 shutdown_connections 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe147423640 1 --2- 172.21.15.129:0/3232661803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cc080 0x7fe14015e010 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.972+0000 7fe147423640 1 --2- 172.21.15.129:0/3232661803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe14006c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.973+0000 7fe147423640 1 -- 172.21.15.129:0/3232661803 >> 172.21.15.129:0/3232661803 conn(0x7fe1400cfb20 msgr2=0x7fe1400cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.973+0000 7fe147423640 1 -- 172.21.15.129:0/3232661803 shutdown_connections 2024-03-23T15:59:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.973+0000 7fe147423640 1 -- 172.21.15.129:0/3232661803 wait complete. 2024-03-23T15:59:30.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.974+0000 7fe147423640 1 Processor -- start 2024-03-23T15:59:30.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.974+0000 7fe147423640 1 -- start start 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe147423640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe1401560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe147423640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 0x7fe1400b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe147423640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1400cc080 0x7fe1400b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe147423640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401f4310 con 0x7fe1400cbc60 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe147423640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401efe50 con 0x7fe1400cc080 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe147423640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe1401f0060 con 0x7fe14006c030 2024-03-23T15:59:30.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe1449be640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 0x7fe1400b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe1449be640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 0x7fe1400b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57492/0 (socket says 172.21.15.129:57492) 2024-03-23T15:59:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe1449be640 1 -- 172.21.15.129:0/2028254569 learned_addr learned my addr 172.21.15.129:0/2028254569 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.976+0000 7fe1451bf640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe1401560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:30.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.977+0000 7fe1459c0640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1400cc080 0x7fe1400b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.980+0000 7fe1449be640 1 -- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 msgr2=0x7fe1401560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.980+0000 7fe1449be640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe1401560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.980+0000 7fe1449be640 1 -- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1400cc080 msgr2=0x7fe1400b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.980+0000 7fe1449be640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1400cc080 0x7fe1400b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:30.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.980+0000 7fe1449be640 1 -- 172.21.15.129:0/2028254569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe138009a70 con 0x7fe1400cbc60 2024-03-23T15:59:30.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.982+0000 7fe1449be640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 0x7fe1400b28f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c400 tx=0x7fe13000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:30.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.986+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130005ce0 con 0x7fe1400cbc60 2024-03-23T15:59:30.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.986+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe130005e40 con 0x7fe1400cbc60 2024-03-23T15:59:30.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.986+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1300072d0 con 0x7fe1400cbc60 2024-03-23T15:59:30.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.986+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1401f0340 con 0x7fe1400cbc60 2024-03-23T15:59:30.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.986+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401f0900 con 0x7fe1400cbc60 2024-03-23T15:59:30.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:30.988+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe140075740 con 0x7fe1400cbc60 2024-03-23T15:59:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.247+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe1300074f0 con 0x7fe1400cbc60 2024-03-23T15:59:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.247+0000 7fe135ffb640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe12406b810 0x7fe12406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.248+0000 7fe1451bf640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe12406b810 0x7fe12406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.248+0000 7fe1451bf640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe12406b810 0x7fe12406dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009aa0 tx=0x7fe13c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.264+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe130058410 con 0x7fe1400cbc60 2024-03-23T15:59:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.264+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe1300546b0 con 0x7fe1400cbc60 2024-03-23T15:59:31.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.501+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1400d0a30 con 0x7fe1400cbc60 2024-03-23T15:59:31.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.525+0000 7fe135ffb640 1 -- 172.21.15.129:0/2028254569 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe130017070 con 0x7fe1400cbc60 2024-03-23T15:59:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:31.528 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:31.528 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:31.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.527+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe12406b810 msgr2=0x7fe12406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:31.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.527+0000 7fe147423640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe12406b810 0x7fe12406dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009aa0 tx=0x7fe13c009190 comp rx=0 tx=0).stop 2024-03-23T15:59:31.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.527+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 msgr2=0x7fe1400b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:31.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.527+0000 7fe147423640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 0x7fe1400b28f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c400 tx=0x7fe13000c8c0 comp rx=0 tx=0).stop 2024-03-23T15:59:31.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 shutdown_connections 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe12406b810 0x7fe12406dcd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1400cc080 0x7fe1400b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1400cbc60 0x7fe1400b28f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 --2- 172.21.15.129:0/2028254569 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe14006c030 0x7fe1401560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 >> 172.21.15.129:0/2028254569 conn(0x7fe1400cfb20 msgr2=0x7fe1400c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 shutdown_connections 2024-03-23T15:59:31.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:31.528+0000 7fe147423640 1 -- 172.21.15.129:0/2028254569 wait complete. 2024-03-23T15:59:32.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:32.146 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:33.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.247+0000 7f6de1781640 1 Processor -- start 2024-03-23T15:59:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.247+0000 7f6de1781640 1 -- start start 2024-03-23T15:59:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.248+0000 7f6de1781640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ccd60 0x7f6ddc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.248+0000 7f6de1781640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ddc0cef40 0x7f6ddc15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.248+0000 7f6de1781640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ddc15b290 con 0x7f6ddc0ceb00 2024-03-23T15:59:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.248+0000 7f6de1781640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f6ddc0ce800 con 0x7f6ddc0cef40 2024-03-23T15:59:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.248+0000 7f6de1781640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc154a70 con 0x7f6ddc0ccd60 2024-03-23T15:59:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.248+0000 7f6ddb7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f6ddc0cef40 0x7f6ddc15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.249+0000 7f6dd3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6ddc0ceb00 0x7f6ddc06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:49424/0 (socket says 172.21.15.129:49424) 2024-03-23T15:59:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.249+0000 7f6dd3fff640 1 -- 172.21.15.129:0/2324354733 learned_addr learned my addr 172.21.15.129:0/2324354733 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.249+0000 7f6ddaffd640 1 --2- 172.21.15.129:0/2324354733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ccd60 0x7f6ddc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.251+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/2324354733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ccd60 msgr2=0x7f6ddc15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.251+0000 7f6ddb7fe640 1 --2- 172.21.15.129:0/2324354733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ccd60 0x7f6ddc15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.251+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/2324354733 >> v1:172.21.15.157:6789/0 conn(0x7f6ddc0ceb00 legacy=0x7f6ddc06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.251+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/2324354733 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc0cd950 con 0x7f6ddc0cef40 2024-03-23T15:59:33.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.253+0000 7f6ddb7fe640 1 --2- 172.21.15.129:0/2324354733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc15b8f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a39fd9b818ba711 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.256+0000 7f6dda7fc640 1 -- 172.21.15.129:0/2324354733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd402cce0 con 0x7f6ddc0cef40 2024-03-23T15:59:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.256+0000 7f6dda7fc640 1 -- 172.21.15.129:0/2324354733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd402ce40 con 0x7f6ddc0cef40 2024-03-23T15:59:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.256+0000 7f6dda7fc640 1 -- 172.21.15.129:0/2324354733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4034660 con 0x7f6ddc0cef40 2024-03-23T15:59:33.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 -- 172.21.15.129:0/2324354733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 msgr2=0x7f6ddc15b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:33.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 --2- 172.21.15.129:0/2324354733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc15b8f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd402bf30 comp rx=0 tx=0).stop 2024-03-23T15:59:33.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 -- 172.21.15.129:0/2324354733 shutdown_connections 2024-03-23T15:59:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 --2- 172.21.15.129:0/2324354733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc15b8f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 --2- 172.21.15.129:0/2324354733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ccd60 0x7f6ddc15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 -- 172.21.15.129:0/2324354733 >> 172.21.15.129:0/2324354733 conn(0x7f6ddc0b1840 msgr2=0x7f6ddc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 -- 172.21.15.129:0/2324354733 shutdown_connections 2024-03-23T15:59:33.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.257+0000 7f6de1781640 1 -- 172.21.15.129:0/2324354733 wait complete. 2024-03-23T15:59:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.259+0000 7f6de1781640 1 Processor -- start 2024-03-23T15:59:33.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.259+0000 7f6de1781640 1 -- start start 2024-03-23T15:59:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.260+0000 7f6de1781640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ddc0ccd60 0x7f6ddc1e5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.260+0000 7f6de1781640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ceb00 0x7f6ddc1e3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.260+0000 7f6de1781640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.260+0000 7f6de1781640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc0c4d40 con 0x7f6ddc0cef40 2024-03-23T15:59:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.260+0000 7f6de1781640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc1e3a80 con 0x7f6ddc0ccd60 2024-03-23T15:59:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.260+0000 7f6de1781640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc1e5c00 con 0x7f6ddc0ceb00 2024-03-23T15:59:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.261+0000 7f6ddb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.261+0000 7f6dd3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ceb00 0x7f6ddc1e3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.261+0000 7f6ddaffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ddc0ccd60 0x7f6ddc1e5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.261+0000 7f6ddb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc151e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57508/0 (socket says 172.21.15.129:57508) 2024-03-23T15:59:33.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.261+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/1363175761 learned_addr learned my addr 172.21.15.129:0/1363175761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:33.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.264+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ceb00 msgr2=0x7f6ddc1e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:33.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.264+0000 7f6ddb7fe640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ceb00 0x7f6ddc1e3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.264+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ddc0ccd60 msgr2=0x7f6ddc1e5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:33.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.264+0000 7f6ddb7fe640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ddc0ccd60 0x7f6ddc1e5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.264+0000 7f6ddb7fe640 1 -- 172.21.15.129:0/1363175761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd4009a70 con 0x7f6ddc0cef40 2024-03-23T15:59:33.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.267+0000 7f6ddb7fe640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc151e00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009eb0 tx=0x7f6dd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.269+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd403d600 con 0x7f6ddc0cef40 2024-03-23T15:59:33.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.269+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ddc1e5ec0 con 0x7f6ddc0cef40 2024-03-23T15:59:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.269+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ddc0c4f90 con 0x7f6ddc0cef40 2024-03-23T15:59:33.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.270+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ddc04f530 con 0x7f6ddc0cef40 2024-03-23T15:59:33.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.272+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd403d760 con 0x7f6ddc0cef40 2024-03-23T15:59:33.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.548+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd403dad0 con 0x7f6ddc0cef40 2024-03-23T15:59:33.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.549+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6dd403b040 con 0x7f6ddc0cef40 2024-03-23T15:59:33.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.549+0000 7f6dd37fe640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6dbc06b840 0x7f6dbc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:33.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.550+0000 7f6ddaffd640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6dbc06b840 0x7f6dbc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:33.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.550+0000 7f6ddaffd640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6dbc06b840 0x7f6dbc06dd00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009b40 tx=0x7f6dcc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:33.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.579+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f6dd407b530 con 0x7f6ddc0cef40 2024-03-23T15:59:33.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.579+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6dd40777d0 con 0x7f6ddc0cef40 2024-03-23T15:59:33.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.805+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ddc0686f0 con 0x7f6ddc0cef40 2024-03-23T15:59:33.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.817+0000 7f6dd37fe640 1 -- 172.21.15.129:0/1363175761 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f6dd404e070 con 0x7f6ddc0cef40 2024-03-23T15:59:33.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:33.820 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:33.820 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6dbc06b840 msgr2=0x7f6dbc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6dbc06b840 0x7f6dbc06dd00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009b40 tx=0x7f6dcc009190 comp rx=0 tx=0).stop 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 msgr2=0x7f6ddc151e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc151e00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009eb0 tx=0x7f6dd4004a40 comp rx=0 tx=0).stop 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 shutdown_connections 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0cef40 0x7f6ddc151e00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ddc0ceb00 0x7f6ddc1e3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f6dbc06b840 0x7f6dbc06dd00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 --2- 172.21.15.129:0/1363175761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ddc0ccd60 0x7f6ddc1e5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 >> 172.21.15.129:0/1363175761 conn(0x7f6ddc0b1840 msgr2=0x7f6ddc1540c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 shutdown_connections 2024-03-23T15:59:33.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:33.820+0000 7f6de1781640 1 -- 172.21.15.129:0/1363175761 wait complete. 2024-03-23T15:59:34.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:34.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:35.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.538+0000 7f045fcaf640 1 Processor -- start 2024-03-23T15:59:35.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.538+0000 7f045fcaf640 1 -- start start 2024-03-23T15:59:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.539+0000 7f045fcaf640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04580ccd60 0x7f045815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.539+0000 7f045fcaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580cef40 0x7f045815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.539+0000 7f045fcaf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045815b290 con 0x7f04580ceb00 2024-03-23T15:59:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.539+0000 7f045fcaf640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f04580ce800 con 0x7f04580ccd60 2024-03-23T15:59:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.539+0000 7f045fcaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0458154a70 con 0x7f04580cef40 2024-03-23T15:59:35.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.540+0000 7f045e24c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580cef40 0x7f045815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:35.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.540+0000 7f045da4b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f04580ccd60 0x7f045815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:35.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.540+0000 7f045d24a640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f04580ceb00 0x7f045806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:59646/0 (socket says 172.21.15.129:59646) 2024-03-23T15:59:35.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.540+0000 7f045d24a640 1 -- 172.21.15.129:0/4188646367 learned_addr learned my addr 172.21.15.129:0/4188646367 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:35.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.543+0000 7f045e24c640 1 -- 172.21.15.129:0/4188646367 >> v2:172.21.15.129:3300/0 conn(0x7f04580ccd60 msgr2=0x7f045815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:35.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.543+0000 7f045e24c640 1 --2- 172.21.15.129:0/4188646367 >> v2:172.21.15.129:3300/0 conn(0x7f04580ccd60 0x7f045815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:35.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.543+0000 7f045e24c640 1 -- 172.21.15.129:0/4188646367 >> v1:172.21.15.129:6789/0 conn(0x7f04580ceb00 legacy=0x7f045806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:35.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.543+0000 7f045e24c640 1 -- 172.21.15.129:0/4188646367 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04580cd950 con 0x7f04580cef40 2024-03-23T15:59:35.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.545+0000 7f045e24c640 1 --2- 172.21.15.129:0/4188646367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580cef40 0x7f045815b8f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0450009d90 tx=0x7f045002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f70e2325a63a930 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:35.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.547+0000 7f045ca49640 1 -- 172.21.15.129:0/4188646367 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045002cce0 con 0x7f04580cef40 2024-03-23T15:59:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.550+0000 7f045ca49640 1 -- 172.21.15.129:0/4188646367 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f045002ce40 con 0x7f04580cef40 2024-03-23T15:59:35.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.550+0000 7f045ca49640 1 -- 172.21.15.129:0/4188646367 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450034660 con 0x7f04580cef40 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 -- 172.21.15.129:0/4188646367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580cef40 msgr2=0x7f045815b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 --2- 172.21.15.129:0/4188646367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580cef40 0x7f045815b8f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0450009d90 tx=0x7f045002bf30 comp rx=0 tx=0).stop 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 -- 172.21.15.129:0/4188646367 shutdown_connections 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 --2- 172.21.15.129:0/4188646367 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580cef40 0x7f045815b8f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 --2- 172.21.15.129:0/4188646367 >> v2:172.21.15.129:3300/0 conn(0x7f04580ccd60 0x7f045815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 -- 172.21.15.129:0/4188646367 >> 172.21.15.129:0/4188646367 conn(0x7f04580b1840 msgr2=0x7f04580b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:35.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 -- 172.21.15.129:0/4188646367 shutdown_connections 2024-03-23T15:59:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.551+0000 7f045fcaf640 1 -- 172.21.15.129:0/4188646367 wait complete. 2024-03-23T15:59:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.553+0000 7f045fcaf640 1 Processor -- start 2024-03-23T15:59:35.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.553+0000 7f045fcaf640 1 -- start start 2024-03-23T15:59:35.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045fcaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580ccd60 0x7f04581e4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045fcaf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 0x7f0458159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045fcaf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04580cef40 0x7f04581f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045fcaf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04581e6060 con 0x7f04580ceb00 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045fcaf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04581f1ae0 con 0x7f04580cef40 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045fcaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04581e50d0 con 0x7f04580ccd60 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045d24a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 0x7f0458159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.554+0000 7f045da4b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580ccd60 0x7f04581e4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.555+0000 7f045d24a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 0x7f0458159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57530/0 (socket says 172.21.15.129:57530) 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.555+0000 7f045d24a640 1 -- 172.21.15.129:0/2929190386 learned_addr learned my addr 172.21.15.129:0/2929190386 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:35.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.555+0000 7f045e24c640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04580cef40 0x7f04581f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:35.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.558+0000 7f045d24a640 1 -- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580ccd60 msgr2=0x7f04581e4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:35.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.558+0000 7f045d24a640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580ccd60 0x7f04581e4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:35.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.558+0000 7f045d24a640 1 -- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04580cef40 msgr2=0x7f04581f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:35.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.558+0000 7f045d24a640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04580cef40 0x7f04581f1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:35.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.558+0000 7f045d24a640 1 -- 172.21.15.129:0/2929190386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0450009a70 con 0x7f04580ceb00 2024-03-23T15:59:35.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.560+0000 7f045d24a640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 0x7f0458159ff0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f044800c3f0 tx=0x7f044800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.564+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448005ce0 con 0x7f04580ceb00 2024-03-23T15:59:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.564+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448005e40 con 0x7f04580ceb00 2024-03-23T15:59:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.564+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480072f0 con 0x7f04580ceb00 2024-03-23T15:59:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.564+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0458155000 con 0x7f04580ceb00 2024-03-23T15:59:35.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.564+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04581e5640 con 0x7f04580ceb00 2024-03-23T15:59:35.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.565+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f045804f530 con 0x7f04580ceb00 2024-03-23T15:59:35.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.820+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0448007450 con 0x7f04580ceb00 2024-03-23T15:59:35.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.820+0000 7f044e7fc640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f043c06b840 0x7f043c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.821+0000 7f045da4b640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f043c06b840 0x7f043c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.821+0000 7f045da4b640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f043c06b840 0x7f043c06dd00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0454009aa0 tx=0x7f0454009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:35.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.838+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f04480581a0 con 0x7f04580ceb00 2024-03-23T15:59:35.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:35.838+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0448054440 con 0x7f04580ceb00 2024-03-23T15:59:36.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.073+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f045806db50 con 0x7f04580ceb00 2024-03-23T15:59:36.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.085+0000 7f044e7fc640 1 -- 172.21.15.129:0/2929190386 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0448017070 con 0x7f04580ceb00 2024-03-23T15:59:36.087 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:36.087 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:36.087 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:36.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f043c06b840 msgr2=0x7f043c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f043c06b840 0x7f043c06dd00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0454009aa0 tx=0x7f0454009190 comp rx=0 tx=0).stop 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 msgr2=0x7f0458159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 0x7f0458159ff0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f044800c3f0 tx=0x7f044800c8b0 comp rx=0 tx=0).stop 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 shutdown_connections 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f043c06b840 0x7f043c06dd00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04580cef40 0x7f04581f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04580ceb00 0x7f0458159ff0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 --2- 172.21.15.129:0/2929190386 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04580ccd60 0x7f04581e4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 >> 172.21.15.129:0/2929190386 conn(0x7f04580b1840 msgr2=0x7f0458154030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 shutdown_connections 2024-03-23T15:59:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:36.087+0000 7f045fcaf640 1 -- 172.21.15.129:0/2929190386 wait complete. 2024-03-23T15:59:36.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:36.705 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:37.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.806+0000 7f3caafd7640 1 Processor -- start 2024-03-23T15:59:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 -- start start 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3ca4154270 0x7f3ca40ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415ad10 0x7f3ca406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ca406c650 0x7f3ca40c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3ca415d880 con 0x7f3ca4154270 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3ca40c4ae0 con 0x7f3ca406c650 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.807+0000 7f3caafd7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca40cd5e0 con 0x7f3ca415ad10 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.808+0000 7f3ca8d73640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3ca4154270 0x7f3ca40ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:37.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.808+0000 7f3ca9574640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ca406c650 0x7f3ca40c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:37.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.808+0000 7f3ca8d73640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f3ca4154270 0x7f3ca40ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57536/0 (socket says 172.21.15.129:57536) 2024-03-23T15:59:37.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.808+0000 7f3ca8d73640 1 -- 172.21.15.129:0/892875306 learned_addr learned my addr 172.21.15.129:0/892875306 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:37.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.808+0000 7f3ca3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415ad10 0x7f3ca406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:37.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.811+0000 7f3ca8d73640 1 -- 172.21.15.129:0/892875306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415ad10 msgr2=0x7f3ca406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:37.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.811+0000 7f3ca8d73640 1 --2- 172.21.15.129:0/892875306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415ad10 0x7f3ca406c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.811+0000 7f3ca8d73640 1 -- 172.21.15.129:0/892875306 >> v2:172.21.15.157:3300/0 conn(0x7f3ca406c650 msgr2=0x7f3ca40c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:37.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.811+0000 7f3ca8d73640 1 --2- 172.21.15.129:0/892875306 >> v2:172.21.15.157:3300/0 conn(0x7f3ca406c650 0x7f3ca40c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.811+0000 7f3ca8d73640 1 -- 172.21.15.129:0/892875306 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca415dab0 con 0x7f3ca4154270 2024-03-23T15:59:37.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.813+0000 7f3ca8d73640 1 --2- 172.21.15.129:0/892875306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca4154270 0x7f3ca40ca580 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40b3050 tx=0x7f3c9802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46be336f474dfa3f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.816+0000 7f3ca37fe640 1 -- 172.21.15.129:0/892875306 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98031750 con 0x7f3ca4154270 2024-03-23T15:59:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.816+0000 7f3ca37fe640 1 -- 172.21.15.129:0/892875306 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c980318b0 con 0x7f3ca4154270 2024-03-23T15:59:37.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.816+0000 7f3ca37fe640 1 -- 172.21.15.129:0/892875306 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98031c40 con 0x7f3ca4154270 2024-03-23T15:59:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 -- 172.21.15.129:0/892875306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca4154270 msgr2=0x7f3ca40ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 --2- 172.21.15.129:0/892875306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca4154270 0x7f3ca40ca580 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40b3050 tx=0x7f3c9802b7f0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 -- 172.21.15.129:0/892875306 shutdown_connections 2024-03-23T15:59:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 --2- 172.21.15.129:0/892875306 >> v2:172.21.15.157:3300/0 conn(0x7f3ca406c650 0x7f3ca40c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 --2- 172.21.15.129:0/892875306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415ad10 0x7f3ca406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 --2- 172.21.15.129:0/892875306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca4154270 0x7f3ca40ca580 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 -- 172.21.15.129:0/892875306 >> 172.21.15.129:0/892875306 conn(0x7f3ca406d290 msgr2=0x7f3ca4072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:37.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 -- 172.21.15.129:0/892875306 shutdown_connections 2024-03-23T15:59:37.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.817+0000 7f3caafd7640 1 -- 172.21.15.129:0/892875306 wait complete. 2024-03-23T15:59:37.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 Processor -- start 2024-03-23T15:59:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 -- start start 2024-03-23T15:59:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca406c650 0x7f3ca41f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 0x7f3ca40d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:37.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca415ad10 0x7f3ca41efd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41f38e0 con 0x7f3ca406c650 2024-03-23T15:59:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca41f32b0 con 0x7f3ca415ad10 2024-03-23T15:59:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.819+0000 7f3caafd7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca41f4240 con 0x7f3ca4154270 2024-03-23T15:59:37.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.820+0000 7f3ca8d73640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca406c650 0x7f3ca41f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:37.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.820+0000 7f3ca3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 0x7f3ca40d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:37.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.820+0000 7f3ca9574640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca415ad10 0x7f3ca41efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:37.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.820+0000 7f3ca3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 0x7f3ca40d04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:45072/0 (socket says 172.21.15.129:45072) 2024-03-23T15:59:37.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.820+0000 7f3ca3fff640 1 -- 172.21.15.129:0/3729077346 learned_addr learned my addr 172.21.15.129:0/3729077346 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:37.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.820+0000 7f3ca8d73640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca406c650 0x7f3ca41f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57552/0 (socket says 172.21.15.129:57552) 2024-03-23T15:59:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.824+0000 7f3ca3fff640 1 -- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca415ad10 msgr2=0x7f3ca41efd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.824+0000 7f3ca3fff640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca415ad10 0x7f3ca41efd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.824+0000 7f3ca3fff640 1 -- 172.21.15.129:0/3729077346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca406c650 msgr2=0x7f3ca41f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.824+0000 7f3ca3fff640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca406c650 0x7f3ca41f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.824+0000 7f3ca3fff640 1 -- 172.21.15.129:0/3729077346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c980094d0 con 0x7f3ca4154270 2024-03-23T15:59:37.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.826+0000 7f3ca3fff640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 0x7f3ca40d04c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000e8a0 tx=0x7f3c9000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.829+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900094f0 con 0x7f3ca4154270 2024-03-23T15:59:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.829+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca40d0a00 con 0x7f3ca4154270 2024-03-23T15:59:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.829+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca41f0800 con 0x7f3ca4154270 2024-03-23T15:59:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.830+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca4068db0 con 0x7f3ca4154270 2024-03-23T15:59:37.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.831+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c90005990 con 0x7f3ca4154270 2024-03-23T15:59:37.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:37.831+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90005d00 con 0x7f3ca4154270 2024-03-23T15:59:38.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.104+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c9000c940 con 0x7f3ca4154270 2024-03-23T15:59:38.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.104+0000 7f3ca17fa640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c8806b840 0x7f3c8806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:38.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.105+0000 7f3ca8d73640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c8806b840 0x7f3c8806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:38.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.105+0000 7f3ca8d73640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c8806b840 0x7f3c8806dd00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3c98035000 tx=0x7f3c98008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:38.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.108+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3c90059200 con 0x7f3ca4154270 2024-03-23T15:59:38.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.120+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c900554a0 con 0x7f3ca4154270 2024-03-23T15:59:38.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.337+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ca40b23f0 con 0x7f3ca4154270 2024-03-23T15:59:38.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.360+0000 7f3ca17fa640 1 -- 172.21.15.129:0/3729077346 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f3c90021020 con 0x7f3ca4154270 2024-03-23T15:59:38.363 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:38.363 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:38.363 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:38.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c8806b840 msgr2=0x7f3c8806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:38.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c8806b840 0x7f3c8806dd00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3c98035000 tx=0x7f3c98008030 comp rx=0 tx=0).stop 2024-03-23T15:59:38.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 msgr2=0x7f3ca40d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:38.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 0x7f3ca40d04c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000e8a0 tx=0x7f3c9000ed60 comp rx=0 tx=0).stop 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 shutdown_connections 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3c8806b840 0x7f3c8806dd00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca415ad10 0x7f3ca41efd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca4154270 0x7f3ca40d04c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 --2- 172.21.15.129:0/3729077346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ca406c650 0x7f3ca41f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 >> 172.21.15.129:0/3729077346 conn(0x7f3ca406d290 msgr2=0x7f3ca40623e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 shutdown_connections 2024-03-23T15:59:38.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:38.364+0000 7f3caafd7640 1 -- 172.21.15.129:0/3729077346 wait complete. 2024-03-23T15:59:38.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:59:38.975 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T15:59:38.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:38.984 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:39.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.085+0000 7f3777902640 1 Processor -- start 2024-03-23T15:59:40.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.086+0000 7f3777902640 1 -- start start 2024-03-23T15:59:40.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.086+0000 7f3777902640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f37700ccd60 0x7f377015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:40.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.086+0000 7f3777902640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f377015b290 con 0x7f37700ceb00 2024-03-23T15:59:40.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.086+0000 7f3777902640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37700ce800 con 0x7f37700cef40 2024-03-23T15:59:40.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.086+0000 7f3777902640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f3770154a70 con 0x7f37700ccd60 2024-03-23T15:59:40.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.087+0000 7f3775e9f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f37700cef40 0x7f377015b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:49436/0 (socket says 172.21.15.129:49436) 2024-03-23T15:59:40.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.087+0000 7f3775e9f640 1 -- 172.21.15.129:0/877030008 learned_addr learned my addr 172.21.15.129:0/877030008 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:40.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.087+0000 7f377569e640 1 --2- 172.21.15.129:0/877030008 >> v2:172.21.15.129:3300/0 conn(0x7f37700ccd60 0x7f377015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:40.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.090+0000 7f377569e640 1 -- 172.21.15.129:0/877030008 >> v1:172.21.15.157:6789/0 conn(0x7f37700cef40 legacy=0x7f377015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:40.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.090+0000 7f377569e640 1 -- 172.21.15.129:0/877030008 >> v1:172.21.15.129:6789/0 conn(0x7f37700ceb00 legacy=0x7f377006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:40.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.090+0000 7f377569e640 1 -- 172.21.15.129:0/877030008 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37700cd950 con 0x7f37700ccd60 2024-03-23T15:59:40.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.093+0000 7f377569e640 1 --2- 172.21.15.129:0/877030008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700ccd60 0x7f377015ad50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f376c0097a0 tx=0x7f376c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9939bb32557d5cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:40.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.098+0000 7f3767fff640 1 -- 172.21.15.129:0/877030008 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c030610 con 0x7f37700ccd60 2024-03-23T15:59:40.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.098+0000 7f3767fff640 1 -- 172.21.15.129:0/877030008 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f376c030770 con 0x7f37700ccd60 2024-03-23T15:59:40.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.098+0000 7f3767fff640 1 -- 172.21.15.129:0/877030008 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c030b00 con 0x7f37700ccd60 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 -- 172.21.15.129:0/877030008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700ccd60 msgr2=0x7f377015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 --2- 172.21.15.129:0/877030008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700ccd60 0x7f377015ad50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f376c0097a0 tx=0x7f376c02b960 comp rx=0 tx=0).stop 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 -- 172.21.15.129:0/877030008 shutdown_connections 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 --2- 172.21.15.129:0/877030008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700ccd60 0x7f377015ad50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 -- 172.21.15.129:0/877030008 >> 172.21.15.129:0/877030008 conn(0x7f37700b1840 msgr2=0x7f37700b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 -- 172.21.15.129:0/877030008 shutdown_connections 2024-03-23T15:59:40.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.099+0000 7f3777902640 1 -- 172.21.15.129:0/877030008 wait complete. 2024-03-23T15:59:40.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.100+0000 7f3777902640 1 Processor -- start 2024-03-23T15:59:40.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.100+0000 7f3777902640 1 -- start start 2024-03-23T15:59:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3777902640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37700ccd60 0x7f37701573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3777902640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37700ceb00 0x7f37700cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3777902640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 0x7f377014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3777902640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f37701f4430 con 0x7f37700cef40 2024-03-23T15:59:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3777902640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f37701f3560 con 0x7f37700ceb00 2024-03-23T15:59:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3777902640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f37701eff70 con 0x7f37700ccd60 2024-03-23T15:59:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3775e9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 0x7f377014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f3774e9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37700ceb00 0x7f37700cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.101+0000 7f377569e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37700ccd60 0x7f37701573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.102+0000 7f3774e9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37700ceb00 0x7f37700cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:47898/0 (socket says 172.21.15.129:47898) 2024-03-23T15:59:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.102+0000 7f3774e9d640 1 -- 172.21.15.129:0/3082654237 learned_addr learned my addr 172.21.15.129:0/3082654237 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:40.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.102+0000 7f3775e9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 0x7f377014ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57570/0 (socket says 172.21.15.129:57570) 2024-03-23T15:59:40.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.105+0000 7f3775e9f640 1 -- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37700ccd60 msgr2=0x7f37701573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:40.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.105+0000 7f3775e9f640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37700ccd60 0x7f37701573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.105+0000 7f3775e9f640 1 -- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37700ceb00 msgr2=0x7f37700cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:40.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.105+0000 7f3775e9f640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37700ceb00 0x7f37700cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.105+0000 7f3775e9f640 1 -- 172.21.15.129:0/3082654237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c009400 con 0x7f37700cef40 2024-03-23T15:59:40.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.108+0000 7f3775e9f640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 0x7f377014ae30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f376800caa0 tx=0x7f376800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.113+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376800ac00 con 0x7f37700cef40 2024-03-23T15:59:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.113+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f376800ad60 con 0x7f37700cef40 2024-03-23T15:59:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.113+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37701f01d0 con 0x7f37700cef40 2024-03-23T15:59:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.113+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768012600 con 0x7f37700cef40 2024-03-23T15:59:40.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.113+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37701f37b0 con 0x7f37700cef40 2024-03-23T15:59:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.114+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f377004f530 con 0x7f37700cef40 2024-03-23T15:59:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.376+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3768007070 con 0x7f37700cef40 2024-03-23T15:59:40.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.376+0000 7f3765ffb640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f375406b840 0x7f375406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:40.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.377+0000 7f377569e640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f375406b840 0x7f375406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:40.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.377+0000 7f377569e640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f375406b840 0x7f375406dd00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f376c036000 tx=0x7f376c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:40.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.399+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3768058c90 con 0x7f37700cef40 2024-03-23T15:59:40.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.399+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3768054f30 con 0x7f37700cef40 2024-03-23T15:59:40.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.618+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37700686f0 con 0x7f37700cef40 2024-03-23T15:59:40.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.651+0000 7f3765ffb640 1 -- 172.21.15.129:0/3082654237 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f37680585f0 con 0x7f37700cef40 2024-03-23T15:59:40.653 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:40.653 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f375406b840 msgr2=0x7f375406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f375406b840 0x7f375406dd00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f376c036000 tx=0x7f376c03b040 comp rx=0 tx=0).stop 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 msgr2=0x7f377014ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 0x7f377014ae30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f376800caa0 tx=0x7f376800cf60 comp rx=0 tx=0).stop 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 shutdown_connections 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37700cef40 0x7f377014ae30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f375406b840 0x7f375406dd00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37700ceb00 0x7f37700cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 --2- 172.21.15.129:0/3082654237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f37700ccd60 0x7f37701573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.653+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 >> 172.21.15.129:0/3082654237 conn(0x7f37700b1840 msgr2=0x7f37700b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:40.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.654+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 shutdown_connections 2024-03-23T15:59:40.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:40.654+0000 7f3777902640 1 -- 172.21.15.129:0/3082654237 wait complete. 2024-03-23T15:59:41.271 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:41.271 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:42.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:42.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.372+0000 7fdd755c7640 1 Processor -- start 2024-03-23T15:59:42.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.372+0000 7fdd755c7640 1 -- start start 2024-03-23T15:59:42.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.373+0000 7fdd755c7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdd7015a2e0 0x7fdd7006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.373+0000 7fdd755c7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd700cbcb0 con 0x7fdd701544b0 2024-03-23T15:59:42.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.373+0000 7fdd755c7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd7015b280 con 0x7fdd7006c0b0 2024-03-23T15:59:42.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.373+0000 7fdd755c7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdd700c5be0 con 0x7fdd7015a2e0 2024-03-23T15:59:42.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.374+0000 7fdd6effd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fdd701544b0 0x7fdd700caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60660/0 (socket says 172.21.15.129:60660) 2024-03-23T15:59:42.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.374+0000 7fdd6effd640 1 -- 172.21.15.129:0/2966258749 learned_addr learned my addr 172.21.15.129:0/2966258749 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:42.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.374+0000 7fdd6e7fc640 1 --2- 172.21.15.129:0/2966258749 >> v2:172.21.15.157:3300/0 conn(0x7fdd7015a2e0 0x7fdd7006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.377+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399602116 0 0) 0x7fdd700cbcb0 con 0x7fdd701544b0 2024-03-23T15:59:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.377+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd58003520 con 0x7fdd701544b0 2024-03-23T15:59:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 <== mon.0 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 451790944 0 0) 0x7fdd58003520 con 0x7fdd701544b0 2024-03-23T15:59:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 >> v2:172.21.15.157:3300/0 conn(0x7fdd7015a2e0 msgr2=0x7fdd7006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 --2- 172.21.15.129:0/2966258749 >> v2:172.21.15.157:3300/0 conn(0x7fdd7015a2e0 0x7fdd7006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 >> v1:172.21.15.157:6789/0 conn(0x7fdd7006c0b0 legacy=0x7fdd700c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7015a9e0 con 0x7fdd701544b0 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 <== mon.0 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3875959834 0 0) 0x7fdd60001670 con 0x7fdd701544b0 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 >> v1:172.21.15.129:6789/0 conn(0x7fdd701544b0 legacy=0x7fdd700caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd58003970 0x7fdd58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd580047c0 0x7fdd58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd58005180 0x7fdd5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd700c5be0 con 0x7fdd580047c0 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5800c030 con 0x7fdd58005180 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.380+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd5800c1f0 con 0x7fdd58003970 2024-03-23T15:59:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.381+0000 7fdd6effd640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd58003970 0x7fdd58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.382+0000 7fdd6e7fc640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd580047c0 0x7fdd58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.384+0000 7fdd6effd640 1 -- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd58005180 msgr2=0x7fdd5800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.384+0000 7fdd6effd640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd58005180 0x7fdd5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.384+0000 7fdd6effd640 1 -- 172.21.15.129:0/2966258749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd580047c0 msgr2=0x7fdd58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.384+0000 7fdd6effd640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd580047c0 0x7fdd58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.384+0000 7fdd6effd640 1 -- 172.21.15.129:0/2966258749 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd58003520 con 0x7fdd58003970 2024-03-23T15:59:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.384+0000 7fdd6f7fe640 1 -- 172.21.15.129:0/2966258749 reap_dead start 2024-03-23T15:59:42.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.386+0000 7fdd6effd640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd58003970 0x7fdd58003d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000c1b0 tx=0x7fdd580057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a7f1a9af35bc11b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.390+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd60005ad0 con 0x7fdd58003970 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.390+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd60005c30 con 0x7fdd58003970 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.390+0000 7fdd6dffb640 1 -- 172.21.15.129:0/2966258749 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6002c870 con 0x7fdd58003970 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 -- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd58003970 msgr2=0x7fdd58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd58003970 0x7fdd58003d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000c1b0 tx=0x7fdd580057f0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 -- 172.21.15.129:0/2966258749 shutdown_connections 2024-03-23T15:59:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 --2- 172.21.15.129:0/2966258749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd58003970 0x7fdd58003d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 -- 172.21.15.129:0/2966258749 >> 172.21.15.129:0/2966258749 conn(0x7fdd70061b50 msgr2=0x7fdd70061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 -- 172.21.15.129:0/2966258749 shutdown_connections 2024-03-23T15:59:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.391+0000 7fdd755c7640 1 -- 172.21.15.129:0/2966258749 wait complete. 2024-03-23T15:59:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.392+0000 7fdd755c7640 1 Processor -- start 2024-03-23T15:59:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.392+0000 7fdd755c7640 1 -- start start 2024-03-23T15:59:42.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.393+0000 7fdd755c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd58003970 0x7fdd700c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.393+0000 7fdd755c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd700c98c0 0x7fdd700b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.393+0000 7fdd755c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd701461a0 0x7fdd70146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.393+0000 7fdd755c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd70145d10 con 0x7fdd58003970 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.393+0000 7fdd755c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd70146b60 con 0x7fdd701461a0 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.393+0000 7fdd755c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd700b44b0 con 0x7fdd700c98c0 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.394+0000 7fdd6e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd700c98c0 0x7fdd700b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.394+0000 7fdd6effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd58003970 0x7fdd700c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.394+0000 7fdd6f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd701461a0 0x7fdd70146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.394+0000 7fdd6e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd700c98c0 0x7fdd700b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49598/0 (socket says 172.21.15.129:49598) 2024-03-23T15:59:42.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.394+0000 7fdd6e7fc640 1 -- 172.21.15.129:0/3666052270 learned_addr learned my addr 172.21.15.129:0/3666052270 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:42.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.397+0000 7fdd6effd640 1 -- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd700c98c0 msgr2=0x7fdd700b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.397+0000 7fdd6effd640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd700c98c0 0x7fdd700b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.397+0000 7fdd6effd640 1 -- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd701461a0 msgr2=0x7fdd70146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.397+0000 7fdd6effd640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd701461a0 0x7fdd70146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.397+0000 7fdd6effd640 1 -- 172.21.15.129:0/3666052270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd700b4790 con 0x7fdd58003970 2024-03-23T15:59:42.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.400+0000 7fdd6effd640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd58003970 0x7fdd700c8670 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800c380 tx=0x7fdd600018f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:42.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.405+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd60001670 con 0x7fdd58003970 2024-03-23T15:59:42.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.405+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd6003a070 con 0x7fdd58003970 2024-03-23T15:59:42.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.405+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd60004a40 con 0x7fdd58003970 2024-03-23T15:59:42.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.405+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd700b4a70 con 0x7fdd58003970 2024-03-23T15:59:42.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.405+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd70147840 con 0x7fdd58003970 2024-03-23T15:59:42.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.406+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd70075740 con 0x7fdd58003970 2024-03-23T15:59:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.429+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd60032070 con 0x7fdd58003970 2024-03-23T15:59:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.429+0000 7fdd577fe640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd5c06b860 0x7fdd5c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.429+0000 7fdd6e7fc640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd5c06b860 0x7fdd5c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:42.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.430+0000 7fdd6e7fc640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd5c06b860 0x7fdd5c06dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdd70157a70 tx=0x7fdd64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:42.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.432+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdd6007a1e0 con 0x7fdd58003970 2024-03-23T15:59:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.753+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd60076480 con 0x7fdd58003970 2024-03-23T15:59:42.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.976+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd7014e530 con 0x7fdd58003970 2024-03-23T15:59:42.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.988+0000 7fdd577fe640 1 -- 172.21.15.129:0/3666052270 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fdd60044070 con 0x7fdd58003970 2024-03-23T15:59:42.990 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:42.990 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:42.991 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd5c06b860 msgr2=0x7fdd5c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd5c06b860 0x7fdd5c06dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdd70157a70 tx=0x7fdd64008040 comp rx=0 tx=0).stop 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd58003970 msgr2=0x7fdd700c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd58003970 0x7fdd700c8670 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800c380 tx=0x7fdd600018f0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 shutdown_connections 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdd5c06b860 0x7fdd5c06dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd701461a0 0x7fdd70146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd700c98c0 0x7fdd700b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 --2- 172.21.15.129:0/3666052270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd58003970 0x7fdd700c8670 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 >> 172.21.15.129:0/3666052270 conn(0x7fdd70061b50 msgr2=0x7fdd7014a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.990+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 shutdown_connections 2024-03-23T15:59:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:42.991+0000 7fdd755c7640 1 -- 172.21.15.129:0/3666052270 wait complete. 2024-03-23T15:59:43.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:43.608 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:44.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:44.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.710+0000 7f875bc6f640 1 Processor -- start 2024-03-23T15:59:44.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.711+0000 7f875bc6f640 1 -- start start 2024-03-23T15:59:44.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.711+0000 7f875bc6f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f875406c410 0x7f875415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:44.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.711+0000 7f875bc6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 0x7f875415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:44.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.711+0000 7f875bc6f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875406c830 con 0x7f875406c030 2024-03-23T15:59:44.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.711+0000 7f875bc6f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f87540cc370 con 0x7f875406c410 2024-03-23T15:59:44.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.711+0000 7f875bc6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f875406cae0 con 0x7f875415de40 2024-03-23T15:59:44.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.712+0000 7f875a20c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 0x7f875415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:44.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.713+0000 7f875a20c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 0x7f875415e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49604/0 (socket says 172.21.15.129:49604) 2024-03-23T15:59:44.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.713+0000 7f875a20c640 1 -- 172.21.15.129:0/905700867 learned_addr learned my addr 172.21.15.129:0/905700867 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:44.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.714+0000 7f875920a640 1 --2- 172.21.15.129:0/905700867 >> v2:172.21.15.129:3300/0 conn(0x7f875406c410 0x7f875415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:44.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.716+0000 7f875a20c640 1 -- 172.21.15.129:0/905700867 >> v2:172.21.15.129:3300/0 conn(0x7f875406c410 msgr2=0x7f875415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:44.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.716+0000 7f875a20c640 1 --2- 172.21.15.129:0/905700867 >> v2:172.21.15.129:3300/0 conn(0x7f875406c410 0x7f875415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:44.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.716+0000 7f875a20c640 1 -- 172.21.15.129:0/905700867 >> v1:172.21.15.129:6789/0 conn(0x7f875406c030 legacy=0x7f87540cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:44.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.716+0000 7f875a20c640 1 -- 172.21.15.129:0/905700867 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87540cabc0 con 0x7f875415de40 2024-03-23T15:59:44.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.719+0000 7f875a20c640 1 --2- 172.21.15.129:0/905700867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 0x7f875415e2b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f874c009e50 tx=0x7f874c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b16fe12b0a92daa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.722+0000 7f8758a09640 1 -- 172.21.15.129:0/905700867 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c02ca70 con 0x7f875415de40 2024-03-23T15:59:44.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.723+0000 7f8758a09640 1 -- 172.21.15.129:0/905700867 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c02cbd0 con 0x7f875415de40 2024-03-23T15:59:44.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.723+0000 7f8758a09640 1 -- 172.21.15.129:0/905700867 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c0343f0 con 0x7f875415de40 2024-03-23T15:59:44.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 -- 172.21.15.129:0/905700867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 msgr2=0x7f875415e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:44.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 --2- 172.21.15.129:0/905700867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 0x7f875415e2b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f874c009e50 tx=0x7f874c02bed0 comp rx=0 tx=0).stop 2024-03-23T15:59:44.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 -- 172.21.15.129:0/905700867 shutdown_connections 2024-03-23T15:59:44.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 --2- 172.21.15.129:0/905700867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875415de40 0x7f875415e2b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:44.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 --2- 172.21.15.129:0/905700867 >> v2:172.21.15.129:3300/0 conn(0x7f875406c410 0x7f875415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:44.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 -- 172.21.15.129:0/905700867 >> 172.21.15.129:0/905700867 conn(0x7f87540cfb20 msgr2=0x7f87540cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:44.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 -- 172.21.15.129:0/905700867 shutdown_connections 2024-03-23T15:59:44.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.724+0000 7f875bc6f640 1 -- 172.21.15.129:0/905700867 wait complete. 2024-03-23T15:59:44.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.725+0000 7f875bc6f640 1 Processor -- start 2024-03-23T15:59:44.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.725+0000 7f875bc6f640 1 -- start start 2024-03-23T15:59:44.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875bc6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f875406c030 0x7f87540b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875bc6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875406c410 0x7f87540b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875bc6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f875415de40 0x7f87540c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875bc6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87540b4eb0 con 0x7f875415de40 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875bc6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87541f0ec0 con 0x7f875406c030 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875bc6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87541f0940 con 0x7f875406c410 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.726+0000 7f875920a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875406c410 0x7f87540b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.727+0000 7f875a20c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f875415de40 0x7f87540c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.727+0000 7f8759a0b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f875406c030 0x7f87540b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:44.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.727+0000 7f875920a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875406c410 0x7f87540b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49616/0 (socket says 172.21.15.129:49616) 2024-03-23T15:59:44.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.727+0000 7f875920a640 1 -- 172.21.15.129:0/4236892656 learned_addr learned my addr 172.21.15.129:0/4236892656 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:44.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.730+0000 7f875a20c640 1 -- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875406c410 msgr2=0x7f87540b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:44.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.730+0000 7f875a20c640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875406c410 0x7f87540b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:44.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.730+0000 7f875a20c640 1 -- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f875406c030 msgr2=0x7f87540b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:44.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.730+0000 7f875a20c640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f875406c030 0x7f87540b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:44.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.730+0000 7f875a20c640 1 -- 172.21.15.129:0/4236892656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c009b30 con 0x7f875415de40 2024-03-23T15:59:44.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.732+0000 7f875a20c640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f875415de40 0x7f87540c4ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f874c032000 tx=0x7f874c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.737+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c009510 con 0x7f875415de40 2024-03-23T15:59:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.737+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c03e450 con 0x7f875415de40 2024-03-23T15:59:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.737+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87540c57a0 con 0x7f875415de40 2024-03-23T15:59:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.737+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c03e820 con 0x7f875415de40 2024-03-23T15:59:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.737+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87541f0b50 con 0x7f875415de40 2024-03-23T15:59:44.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.738+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87541f0ce0 con 0x7f875415de40 2024-03-23T15:59:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.758+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f874c03e980 con 0x7f875415de40 2024-03-23T15:59:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.758+0000 7f874a7fc640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f873806b860 0x7f873806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:44.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.759+0000 7f8759a0b640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f873806b860 0x7f873806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:44.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:44.759+0000 7f8759a0b640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f873806b860 0x7f873806dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8750006fd0 tx=0x7f8750009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:45.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.027+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f874c07b960 con 0x7f875415de40 2024-03-23T15:59:45.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.027+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f874c077c00 con 0x7f875415de40 2024-03-23T15:59:45.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.303+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87540732f0 con 0x7f875415de40 2024-03-23T15:59:45.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.323+0000 7f874a7fc640 1 -- 172.21.15.129:0/4236892656 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f874c042020 con 0x7f875415de40 2024-03-23T15:59:45.326 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:45.326 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:45.326 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:45.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f873806b860 msgr2=0x7f873806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f873806b860 0x7f873806dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8750006fd0 tx=0x7f8750009060 comp rx=0 tx=0).stop 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f875415de40 msgr2=0x7f87540c4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f875415de40 0x7f87540c4ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f874c032000 tx=0x7f874c004a40 comp rx=0 tx=0).stop 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 shutdown_connections 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f875415de40 0x7f87540c4ee0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f875406c410 0x7f87540b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f873806b860 0x7f873806dd20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 --2- 172.21.15.129:0/4236892656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f875406c030 0x7f87540b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 >> 172.21.15.129:0/4236892656 conn(0x7f87540cfb20 msgr2=0x7f87540ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 shutdown_connections 2024-03-23T15:59:45.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:45.325+0000 7f875bc6f640 1 -- 172.21.15.129:0/4236892656 wait complete. 2024-03-23T15:59:45.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:45.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:46.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:47.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.046+0000 7fcc2553e640 1 Processor -- start 2024-03-23T15:59:47.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.047+0000 7fcc2553e640 1 -- start start 2024-03-23T15:59:47.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.048+0000 7fcc2553e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc201544b0 0x7fcc2015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.048+0000 7fcc2553e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 0x7fcc2006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.048+0000 7fcc2553e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc2006c0b0 0x7fcc200c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.048+0000 7fcc2553e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fcc200c5be0 con 0x7fcc2006c0b0 2024-03-23T15:59:47.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.048+0000 7fcc2553e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcc200cbcb0 con 0x7fcc201544b0 2024-03-23T15:59:47.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.048+0000 7fcc2553e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2015b280 con 0x7fcc200caa00 2024-03-23T15:59:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.049+0000 7fcc1f7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fcc2006c0b0 0x7fcc200c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.049+0000 7fcc1effd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc201544b0 0x7fcc2015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.049+0000 7fcc1e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 0x7fcc2006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.049+0000 7fcc1e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 0x7fcc2006bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49626/0 (socket says 172.21.15.129:49626) 2024-03-23T15:59:47.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.049+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/4231413874 learned_addr learned my addr 172.21.15.129:0/4231413874 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:47.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.052+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/4231413874 >> v2:172.21.15.157:3300/0 conn(0x7fcc201544b0 msgr2=0x7fcc2015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:47.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.052+0000 7fcc1e7fc640 1 --2- 172.21.15.129:0/4231413874 >> v2:172.21.15.157:3300/0 conn(0x7fcc201544b0 0x7fcc2015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.052+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/4231413874 >> v2:172.21.15.129:3300/0 conn(0x7fcc2006c0b0 msgr2=0x7fcc200c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:47.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.052+0000 7fcc1e7fc640 1 --2- 172.21.15.129:0/4231413874 >> v2:172.21.15.129:3300/0 conn(0x7fcc2006c0b0 0x7fcc200c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.052+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/4231413874 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2015a9e0 con 0x7fcc200caa00 2024-03-23T15:59:47.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.055+0000 7fcc1e7fc640 1 --2- 172.21.15.129:0/4231413874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 0x7fcc2006bb70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc2006cd70 tx=0x7fcc1402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42393404ea28a153 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:47.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.060+0000 7fcc1dffb640 1 -- 172.21.15.129:0/4231413874 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14030840 con 0x7fcc200caa00 2024-03-23T15:59:47.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.060+0000 7fcc1dffb640 1 -- 172.21.15.129:0/4231413874 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc140309a0 con 0x7fcc200caa00 2024-03-23T15:59:47.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.060+0000 7fcc1dffb640 1 -- 172.21.15.129:0/4231413874 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14030d30 con 0x7fcc200caa00 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 -- 172.21.15.129:0/4231413874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 msgr2=0x7fcc2006bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 --2- 172.21.15.129:0/4231413874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 0x7fcc2006bb70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc2006cd70 tx=0x7fcc1402b980 comp rx=0 tx=0).stop 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 -- 172.21.15.129:0/4231413874 shutdown_connections 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 --2- 172.21.15.129:0/4231413874 >> v2:172.21.15.129:3300/0 conn(0x7fcc2006c0b0 0x7fcc200c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 --2- 172.21.15.129:0/4231413874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc200caa00 0x7fcc2006bb70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 --2- 172.21.15.129:0/4231413874 >> v2:172.21.15.157:3300/0 conn(0x7fcc201544b0 0x7fcc2015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 -- 172.21.15.129:0/4231413874 >> 172.21.15.129:0/4231413874 conn(0x7fcc20061b50 msgr2=0x7fcc20061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 -- 172.21.15.129:0/4231413874 shutdown_connections 2024-03-23T15:59:47.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.061+0000 7fcc2553e640 1 -- 172.21.15.129:0/4231413874 wait complete. 2024-03-23T15:59:47.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.063+0000 7fcc2553e640 1 Processor -- start 2024-03-23T15:59:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.063+0000 7fcc2553e640 1 -- start start 2024-03-23T15:59:47.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.064+0000 7fcc2553e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2006c0b0 0x7fcc201f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.064+0000 7fcc2553e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 0x7fcc20141910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.064+0000 7fcc2553e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc201544b0 0x7fcc201592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.064+0000 7fcc2553e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc20156db0 con 0x7fcc2006c0b0 2024-03-23T15:59:47.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.064+0000 7fcc2553e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200cee70 con 0x7fcc200caa00 2024-03-23T15:59:47.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.064+0000 7fcc2553e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc201f3f90 con 0x7fcc201544b0 2024-03-23T15:59:47.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.065+0000 7fcc1effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2006c0b0 0x7fcc201f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.065+0000 7fcc1e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 0x7fcc20141910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.065+0000 7fcc1f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc201544b0 0x7fcc201592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.065+0000 7fcc1e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 0x7fcc20141910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:43948/0 (socket says 172.21.15.129:43948) 2024-03-23T15:59:47.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.065+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/1153284330 learned_addr learned my addr 172.21.15.129:0/1153284330 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.070+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc201544b0 msgr2=0x7fcc201592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.070+0000 7fcc1e7fc640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc201544b0 0x7fcc201592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.070+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/1153284330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2006c0b0 msgr2=0x7fcc201f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.070+0000 7fcc1e7fc640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2006c0b0 0x7fcc201f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.070+0000 7fcc1e7fc640 1 -- 172.21.15.129:0/1153284330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140094d0 con 0x7fcc200caa00 2024-03-23T15:59:47.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.070+0000 7fcc1effd640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2006c0b0 0x7fcc201f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:59:47.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.074+0000 7fcc1e7fc640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 0x7fcc20141910 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcc14000c00 tx=0x7fcc14004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.078+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14039c90 con 0x7fcc200caa00 2024-03-23T15:59:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.078+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc201efb20 con 0x7fcc200caa00 2024-03-23T15:59:47.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.078+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc201f00c0 con 0x7fcc200caa00 2024-03-23T15:59:47.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.079+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14039df0 con 0x7fcc200caa00 2024-03-23T15:59:47.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.079+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1404d660 con 0x7fcc200caa00 2024-03-23T15:59:47.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.079+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc00005290 con 0x7fcc200caa00 2024-03-23T15:59:47.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.350+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc14057430 con 0x7fcc200caa00 2024-03-23T15:59:47.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.350+0000 7fcc077fe640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b810 0x7fcc0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:47.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.351+0000 7fcc1effd640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b810 0x7fcc0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:47.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.351+0000 7fcc1effd640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b810 0x7fcc0c06dcd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009e90 tx=0x7fcc10009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:47.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.357+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fcc1403e070 con 0x7fcc200caa00 2024-03-23T15:59:47.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.375+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc14078550 con 0x7fcc200caa00 2024-03-23T15:59:47.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.599+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc000058f0 con 0x7fcc200caa00 2024-03-23T15:59:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.620+0000 7fcc077fe640 1 -- 172.21.15.129:0/1153284330 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fcc000058f0 con 0x7fcc200caa00 2024-03-23T15:59:47.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:47.623 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:47.623 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:47.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b810 msgr2=0x7fcc0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:47.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b810 0x7fcc0c06dcd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009e90 tx=0x7fcc10009190 comp rx=0 tx=0).stop 2024-03-23T15:59:47.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 msgr2=0x7fcc20141910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:47.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 0x7fcc20141910 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcc14000c00 tx=0x7fcc14004730 comp rx=0 tx=0).stop 2024-03-23T15:59:47.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 shutdown_connections 2024-03-23T15:59:47.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc201544b0 0x7fcc201592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fcc0c06b810 0x7fcc0c06dcd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc200caa00 0x7fcc20141910 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 --2- 172.21.15.129:0/1153284330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc2006c0b0 0x7fcc201f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:47.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 >> 172.21.15.129:0/1153284330 conn(0x7fcc20061b50 msgr2=0x7fcc200c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:47.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 shutdown_connections 2024-03-23T15:59:47.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:47.623+0000 7fcc2553e640 1 -- 172.21.15.129:0/1153284330 wait complete. 2024-03-23T15:59:48.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:48.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:49.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.343+0000 7efe15faf640 1 Processor -- start 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.343+0000 7efe15faf640 1 -- start start 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe15faf640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe100ccd60 0x7efe1015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe15faf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efe100cef40 0x7efe1015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe15faf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe1015b290 con 0x7efe100ceb00 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe15faf640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7efe100ce800 con 0x7efe100ccd60 2024-03-23T15:59:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe15faf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efe10154a70 con 0x7efe100cef40 2024-03-23T15:59:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe0f7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe100ccd60 0x7efe1015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.344+0000 7efe0ffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efe100cef40 0x7efe1015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.345+0000 7efe0f7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe100ccd60 0x7efe1015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44744/0 (socket says 172.21.15.129:44744) 2024-03-23T15:59:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.345+0000 7efe0f7fe640 1 -- 172.21.15.129:0/3093515821 learned_addr learned my addr 172.21.15.129:0/3093515821 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:49.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.348+0000 7efe0ffff640 1 -- 172.21.15.129:0/3093515821 >> v2:172.21.15.129:3300/0 conn(0x7efe100ccd60 msgr2=0x7efe1015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:49.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.348+0000 7efe0ffff640 1 --2- 172.21.15.129:0/3093515821 >> v2:172.21.15.129:3300/0 conn(0x7efe100ccd60 0x7efe1015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.348+0000 7efe0ffff640 1 -- 172.21.15.129:0/3093515821 >> v1:172.21.15.129:6789/0 conn(0x7efe100ceb00 legacy=0x7efe1006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:49.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.348+0000 7efe0ffff640 1 -- 172.21.15.129:0/3093515821 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe100cd950 con 0x7efe100cef40 2024-03-23T15:59:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.351+0000 7efe0ffff640 1 --2- 172.21.15.129:0/3093515821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100cef40 0x7efe1015b8f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efe100cf660 tx=0x7efe0802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31b2a2cec6f3b31d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe0e7fc640 1 -- 172.21.15.129:0/3093515821 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08030610 con 0x7efe100cef40 2024-03-23T15:59:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe0e7fc640 1 -- 172.21.15.129:0/3093515821 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe08030770 con 0x7efe100cef40 2024-03-23T15:59:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe0e7fc640 1 -- 172.21.15.129:0/3093515821 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08030b00 con 0x7efe100cef40 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe15faf640 1 -- 172.21.15.129:0/3093515821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100cef40 msgr2=0x7efe1015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe15faf640 1 --2- 172.21.15.129:0/3093515821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100cef40 0x7efe1015b8f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efe100cf660 tx=0x7efe0802b960 comp rx=0 tx=0).stop 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe15faf640 1 -- 172.21.15.129:0/3093515821 shutdown_connections 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe15faf640 1 --2- 172.21.15.129:0/3093515821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100cef40 0x7efe1015b8f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.356+0000 7efe15faf640 1 --2- 172.21.15.129:0/3093515821 >> v2:172.21.15.129:3300/0 conn(0x7efe100ccd60 0x7efe1015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.357+0000 7efe15faf640 1 -- 172.21.15.129:0/3093515821 >> 172.21.15.129:0/3093515821 conn(0x7efe100b1840 msgr2=0x7efe100b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.357+0000 7efe15faf640 1 -- 172.21.15.129:0/3093515821 shutdown_connections 2024-03-23T15:59:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.357+0000 7efe15faf640 1 -- 172.21.15.129:0/3093515821 wait complete. 2024-03-23T15:59:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.358+0000 7efe15faf640 1 Processor -- start 2024-03-23T15:59:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.358+0000 7efe15faf640 1 -- start start 2024-03-23T15:59:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe15faf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe100ccd60 0x7efe101f0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe15faf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100ceb00 0x7efe101f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe15faf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 0x7efe100c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe15faf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe101f1530 con 0x7efe100ccd60 2024-03-23T15:59:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe15faf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efe101f0c90 con 0x7efe100ceb00 2024-03-23T15:59:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe15faf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efe101f0f70 con 0x7efe100cef40 2024-03-23T15:59:49.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe0f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe100ccd60 0x7efe101f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:49.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe0effd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100ceb00 0x7efe101f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:49.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.359+0000 7efe0ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 0x7efe100c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:49.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.360+0000 7efe0ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 0x7efe100c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49636/0 (socket says 172.21.15.129:49636) 2024-03-23T15:59:49.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.360+0000 7efe0ffff640 1 -- 172.21.15.129:0/281163802 learned_addr learned my addr 172.21.15.129:0/281163802 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.363+0000 7efe0ffff640 1 -- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100ceb00 msgr2=0x7efe101f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.363+0000 7efe0ffff640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100ceb00 0x7efe101f4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.363+0000 7efe0ffff640 1 -- 172.21.15.129:0/281163802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe100ccd60 msgr2=0x7efe101f0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.363+0000 7efe0ffff640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe100ccd60 0x7efe101f0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.363+0000 7efe0ffff640 1 -- 172.21.15.129:0/281163802 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe08009400 con 0x7efe100cef40 2024-03-23T15:59:49.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.365+0000 7efe0ffff640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 0x7efe100c6dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe0802bf70 tx=0x7efe08004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:49.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.370+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0803e040 con 0x7efe100cef40 2024-03-23T15:59:49.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.370+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe08039c30 con 0x7efe100cef40 2024-03-23T15:59:49.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.370+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08030e10 con 0x7efe100cef40 2024-03-23T15:59:49.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.370+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe101f1250 con 0x7efe100cef40 2024-03-23T15:59:49.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.370+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe101f3f10 con 0x7efe100cef40 2024-03-23T15:59:49.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.371+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe1004f530 con 0x7efe100cef40 2024-03-23T15:59:49.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.660+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efe080443f0 con 0x7efe100cef40 2024-03-23T15:59:49.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.660+0000 7efdf7fff640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efdfc06b890 0x7efdfc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:49.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.660+0000 7efe0f7fe640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efdfc06b890 0x7efdfc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:49.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.661+0000 7efe0f7fe640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efdfc06b890 0x7efdfc06dd50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efe00011340 tx=0x7efe0000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:49.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.666+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efe0807b4c0 con 0x7efe100cef40 2024-03-23T15:59:49.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.684+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efe08077860 con 0x7efe100cef40 2024-03-23T15:59:49.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.940+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe100686f0 con 0x7efe100cef40 2024-03-23T15:59:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.960+0000 7efdf7fff640 1 -- 172.21.15.129:0/281163802 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7efe0807ae20 con 0x7efe100cef40 2024-03-23T15:59:49.963 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:49.963 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:49.963 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efdfc06b890 msgr2=0x7efdfc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efdfc06b890 0x7efdfc06dd50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efe00011340 tx=0x7efe0000b040 comp rx=0 tx=0).stop 2024-03-23T15:59:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 msgr2=0x7efe100c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 0x7efe100c6dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe0802bf70 tx=0x7efe08004730 comp rx=0 tx=0).stop 2024-03-23T15:59:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 shutdown_connections 2024-03-23T15:59:49.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe100cef40 0x7efe100c6dd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efdfc06b890 0x7efdfc06dd50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe100ceb00 0x7efe101f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 --2- 172.21.15.129:0/281163802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe100ccd60 0x7efe101f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 >> 172.21.15.129:0/281163802 conn(0x7efe100b1840 msgr2=0x7efe101e5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 shutdown_connections 2024-03-23T15:59:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:49.963+0000 7efe15faf640 1 -- 172.21.15.129:0/281163802 wait complete. 2024-03-23T15:59:50.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:50.580 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:51.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:51.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.682+0000 7ff3d0725640 1 Processor -- start 2024-03-23T15:59:51.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.682+0000 7ff3d0725640 1 -- start start 2024-03-23T15:59:51.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.683+0000 7ff3d0725640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 0x7ff3c815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:51.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.683+0000 7ff3d0725640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff3c815de40 0x7ff3c815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:51.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.683+0000 7ff3d0725640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3c806c830 con 0x7ff3c806c030 2024-03-23T15:59:51.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.683+0000 7ff3d0725640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff3c80cc370 con 0x7ff3c815de40 2024-03-23T15:59:51.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.683+0000 7ff3d0725640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c806cae0 con 0x7ff3c806c410 2024-03-23T15:59:51.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.683+0000 7ff3cdcc0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 0x7ff3c815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:51.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.684+0000 7ff3cecc2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff3c815de40 0x7ff3c815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:51.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.684+0000 7ff3cdcc0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 0x7ff3c815d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:36266/0 (socket says 172.21.15.129:36266) 2024-03-23T15:59:51.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.684+0000 7ff3cdcc0640 1 -- 172.21.15.129:0/886588415 learned_addr learned my addr 172.21.15.129:0/886588415 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.686+0000 7ff3cdcc0640 1 -- 172.21.15.129:0/886588415 >> v2:172.21.15.157:3300/0 conn(0x7ff3c815de40 msgr2=0x7ff3c815e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.686+0000 7ff3cdcc0640 1 --2- 172.21.15.129:0/886588415 >> v2:172.21.15.157:3300/0 conn(0x7ff3c815de40 0x7ff3c815e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.686+0000 7ff3cdcc0640 1 -- 172.21.15.129:0/886588415 >> v1:172.21.15.129:6789/0 conn(0x7ff3c806c030 legacy=0x7ff3c80cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.686+0000 7ff3cdcc0640 1 -- 172.21.15.129:0/886588415 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c80cabc0 con 0x7ff3c806c410 2024-03-23T15:59:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.689+0000 7ff3cdcc0640 1 --2- 172.21.15.129:0/886588415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 0x7ff3c815d900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd65cfaa78b5a9f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:51.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.694+0000 7ff3cd4bf640 1 -- 172.21.15.129:0/886588415 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8030840 con 0x7ff3c806c410 2024-03-23T15:59:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.694+0000 7ff3cd4bf640 1 -- 172.21.15.129:0/886588415 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3b80309a0 con 0x7ff3c806c410 2024-03-23T15:59:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.694+0000 7ff3cd4bf640 1 -- 172.21.15.129:0/886588415 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8030d30 con 0x7ff3c806c410 2024-03-23T15:59:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 -- 172.21.15.129:0/886588415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 msgr2=0x7ff3c815d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 --2- 172.21.15.129:0/886588415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 0x7ff3c815d900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009870 tx=0x7ff3b802b980 comp rx=0 tx=0).stop 2024-03-23T15:59:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 -- 172.21.15.129:0/886588415 shutdown_connections 2024-03-23T15:59:51.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 --2- 172.21.15.129:0/886588415 >> v2:172.21.15.157:3300/0 conn(0x7ff3c815de40 0x7ff3c815e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:51.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 --2- 172.21.15.129:0/886588415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c806c410 0x7ff3c815d900 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:51.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 -- 172.21.15.129:0/886588415 >> 172.21.15.129:0/886588415 conn(0x7ff3c80cfb20 msgr2=0x7ff3c80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:51.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 -- 172.21.15.129:0/886588415 shutdown_connections 2024-03-23T15:59:51.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.695+0000 7ff3d0725640 1 -- 172.21.15.129:0/886588415 wait complete. 2024-03-23T15:59:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.696+0000 7ff3d0725640 1 Processor -- start 2024-03-23T15:59:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.696+0000 7ff3d0725640 1 -- start start 2024-03-23T15:59:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3d0725640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c806c030 0x7ff3c8154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3d0725640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c806c410 0x7ff3c8159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3d0725640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c815de40 0x7ff3c81f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:51.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3d0725640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c8159800 con 0x7ff3c806c410 2024-03-23T15:59:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3d0725640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c81f1200 con 0x7ff3c806c030 2024-03-23T15:59:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3d0725640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c8158510 con 0x7ff3c815de40 2024-03-23T15:59:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.697+0000 7ff3cecc2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c815de40 0x7ff3c81f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.698+0000 7ff3cdcc0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c806c410 0x7ff3c8159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.698+0000 7ff3ce4c1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c806c030 0x7ff3c8154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:51.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.698+0000 7ff3cecc2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c815de40 0x7ff3c81f0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:36276/0 (socket says 172.21.15.129:36276) 2024-03-23T15:59:51.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.698+0000 7ff3cecc2640 1 -- 172.21.15.129:0/841779924 learned_addr learned my addr 172.21.15.129:0/841779924 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.702+0000 7ff3ce4c1640 1 -- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c815de40 msgr2=0x7ff3c81f0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.702+0000 7ff3ce4c1640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c815de40 0x7ff3c81f0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.702+0000 7ff3ce4c1640 1 -- 172.21.15.129:0/841779924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c806c410 msgr2=0x7ff3c8159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:51.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.702+0000 7ff3ce4c1640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c806c410 0x7ff3c8159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:51.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.702+0000 7ff3ce4c1640 1 -- 172.21.15.129:0/841779924 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b80094d0 con 0x7ff3c806c030 2024-03-23T15:59:51.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.704+0000 7ff3ce4c1640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c806c030 0x7ff3c8154d00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400c340 tx=0x7ff3c400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:51.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.708+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40052d0 con 0x7ff3c806c030 2024-03-23T15:59:51.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.708+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c4005450 con 0x7ff3c806c030 2024-03-23T15:59:51.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.708+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4007260 con 0x7ff3c806c030 2024-03-23T15:59:51.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.709+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c81587d0 con 0x7ff3c806c030 2024-03-23T15:59:51.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.709+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c8159140 con 0x7ff3c806c030 2024-03-23T15:59:51.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.709+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c8158fa0 con 0x7ff3c806c030 2024-03-23T15:59:51.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.840+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff3c40073c0 con 0x7ff3c806c030 2024-03-23T15:59:51.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.840+0000 7ff3beffd640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff3ac06b860 0x7ff3ac06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:51.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.841+0000 7ff3cdcc0640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff3ac06b860 0x7ff3ac06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:51.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.841+0000 7ff3cdcc0640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff3ac06b860 0x7ff3ac06dd20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8036000 tx=0x7ff3b8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:51.846+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff3c4058070 con 0x7ff3c806c030 2024-03-23T15:59:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.041+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff3c4054310 con 0x7ff3c806c030 2024-03-23T15:59:52.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.259+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3c8158fa0 con 0x7ff3c806c030 2024-03-23T15:59:52.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.275+0000 7ff3beffd640 1 -- 172.21.15.129:0/841779924 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff3c8158fa0 con 0x7ff3c806c030 2024-03-23T15:59:52.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:52.277 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:52.278 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff3ac06b860 msgr2=0x7ff3ac06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff3ac06b860 0x7ff3ac06dd20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8036000 tx=0x7ff3b8008030 comp rx=0 tx=0).stop 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c806c030 msgr2=0x7ff3c8154d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c806c030 0x7ff3c8154d00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400c340 tx=0x7ff3c400c800 comp rx=0 tx=0).stop 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 shutdown_connections 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c815de40 0x7ff3c81f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3c806c410 0x7ff3c8159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff3ac06b860 0x7ff3ac06dd20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 --2- 172.21.15.129:0/841779924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c806c030 0x7ff3c8154d00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 >> 172.21.15.129:0/841779924 conn(0x7ff3c80cfb20 msgr2=0x7ff3c80ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 shutdown_connections 2024-03-23T15:59:52.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:52.277+0000 7ff3d0725640 1 -- 172.21.15.129:0/841779924 wait complete. 2024-03-23T15:59:52.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:52.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:53.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:53.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.996+0000 7f4f2328f640 1 Processor -- start 2024-03-23T15:59:53.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.997+0000 7f4f2328f640 1 -- start start 2024-03-23T15:59:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.998+0000 7f4f2328f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4f1c15e010 0x7f4f1c165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.998+0000 7f4f2328f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f1c06c830 con 0x7f4f1c06c030 2024-03-23T15:59:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.998+0000 7f4f2328f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f1c0cc370 con 0x7f4f1c06c410 2024-03-23T15:59:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.998+0000 7f4f2328f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4f1c06cae0 con 0x7f4f1c15e010 2024-03-23T15:59:54.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.999+0000 7f4f2182c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4f1c15e010 0x7f4f1c165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:54.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.999+0000 7f4f2102b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4f1c06c030 0x7f4f1c0cbc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:44010/0 (socket says 172.21.15.129:44010) 2024-03-23T15:59:54.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:53.999+0000 7f4f2102b640 1 -- 172.21.15.129:0/2887861138 learned_addr learned my addr 172.21.15.129:0/2887861138 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:54.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.002+0000 7f4f13fff640 1 -- 172.21.15.129:0/2887861138 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765463846 0 0) 0x7f4f1c06c830 con 0x7f4f1c06c030 2024-03-23T15:59:54.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.002+0000 7f4f13fff640 1 -- 172.21.15.129:0/2887861138 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f08003520 con 0x7f4f1c06c030 2024-03-23T15:59:54.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.005+0000 7f4f2182c640 1 -- 172.21.15.129:0/2887861138 >> v1:172.21.15.157:6789/0 conn(0x7f4f1c06c410 legacy=0x7f4f1c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.005+0000 7f4f2182c640 1 -- 172.21.15.129:0/2887861138 >> v1:172.21.15.129:6789/0 conn(0x7f4f1c06c030 legacy=0x7f4f1c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.005+0000 7f4f2182c640 1 -- 172.21.15.129:0/2887861138 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1c0cabc0 con 0x7f4f1c15e010 2024-03-23T15:59:54.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.009+0000 7f4f2182c640 1 --2- 172.21.15.129:0/2887861138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c165c60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009d90 tx=0x7f4f1402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b13247eef35adb09 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:54.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.015+0000 7f4f13fff640 1 -- 172.21.15.129:0/2887861138 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1402cd00 con 0x7f4f1c15e010 2024-03-23T15:59:54.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.015+0000 7f4f13fff640 1 -- 172.21.15.129:0/2887861138 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f1402ce60 con 0x7f4f1c15e010 2024-03-23T15:59:54.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.015+0000 7f4f13fff640 1 -- 172.21.15.129:0/2887861138 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14034660 con 0x7f4f1c15e010 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 -- 172.21.15.129:0/2887861138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 msgr2=0x7f4f1c165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2887861138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c165c60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009d90 tx=0x7f4f1402bf50 comp rx=0 tx=0).stop 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 -- 172.21.15.129:0/2887861138 shutdown_connections 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2887861138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c165c60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 -- 172.21.15.129:0/2887861138 >> 172.21.15.129:0/2887861138 conn(0x7f4f1c0cfb20 msgr2=0x7f4f1c0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 -- 172.21.15.129:0/2887861138 shutdown_connections 2024-03-23T15:59:54.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.016+0000 7f4f2328f640 1 -- 172.21.15.129:0/2887861138 wait complete. 2024-03-23T15:59:54.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.017+0000 7f4f2328f640 1 Processor -- start 2024-03-23T15:59:54.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.017+0000 7f4f2328f640 1 -- start start 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2328f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 0x7f4f1c1f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2328f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f1c06c410 0x7f4f1c1f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2328f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c0b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2328f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c157180 con 0x7f4f1c06c030 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2328f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c0b5900 con 0x7f4f1c15e010 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2328f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1c14ea60 con 0x7f4f1c06c410 2024-03-23T15:59:54.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2102b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 0x7f4f1c1f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:54.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2182c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c0b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:54.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.018+0000 7f4f2082a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f1c06c410 0x7f4f1c1f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:54.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.019+0000 7f4f2102b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 0x7f4f1c1f1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43582/0 (socket says 172.21.15.129:43582) 2024-03-23T15:59:54.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.019+0000 7f4f2102b640 1 -- 172.21.15.129:0/2264587668 learned_addr learned my addr 172.21.15.129:0/2264587668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:54.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.022+0000 7f4f2102b640 1 -- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f1c06c410 msgr2=0x7f4f1c1f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.022+0000 7f4f2102b640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f1c06c410 0x7f4f1c1f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.022+0000 7f4f2102b640 1 -- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 msgr2=0x7f4f1c0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.022+0000 7f4f2102b640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c0b53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.022+0000 7f4f2102b640 1 -- 172.21.15.129:0/2264587668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f14009a70 con 0x7f4f1c06c030 2024-03-23T15:59:54.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.025+0000 7f4f2102b640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 0x7f4f1c1f1820 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c440 tx=0x7f4f1800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:54.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.029+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18005ce0 con 0x7f4f1c06c030 2024-03-23T15:59:54.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.029+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f18005e40 con 0x7f4f1c06c030 2024-03-23T15:59:54.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.029+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1c150fc0 con 0x7f4f1c06c030 2024-03-23T15:59:54.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.029+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f180072b0 con 0x7f4f1c06c030 2024-03-23T15:59:54.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.029+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1c151580 con 0x7f4f1c06c030 2024-03-23T15:59:54.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.030+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f1c0686f0 con 0x7f4f1c06c030 2024-03-23T15:59:54.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.289+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f18007410 con 0x7f4f1c06c030 2024-03-23T15:59:54.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.289+0000 7f4f11ffb640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f0006b840 0x7f4f0006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:54.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.289+0000 7f4f2082a640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f0006b840 0x7f4f0006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:54.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.290+0000 7f4f2082a640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f0006b840 0x7f4f0006dd00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c154370 tx=0x7f4f0c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:54.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.310+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f4f18058270 con 0x7f4f1c06c030 2024-03-23T15:59:54.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.310+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f18058040 con 0x7f4f1c06c030 2024-03-23T15:59:54.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.526+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f1c06bbe0 con 0x7f4f1c06c030 2024-03-23T15:59:54.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.547+0000 7f4f11ffb640 1 -- 172.21.15.129:0/2264587668 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f4f18017020 con 0x7f4f1c06c030 2024-03-23T15:59:54.549 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:54.549 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:54.550 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f0006b840 msgr2=0x7f4f0006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f0006b840 0x7f4f0006dd00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c154370 tx=0x7f4f0c009040 comp rx=0 tx=0).stop 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 msgr2=0x7f4f1c1f1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 0x7f4f1c1f1820 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c440 tx=0x7f4f1800c900 comp rx=0 tx=0).stop 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 shutdown_connections 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f4f0006b840 0x7f4f0006dd00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f1c15e010 0x7f4f1c0b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f1c06c410 0x7f4f1c1f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 --2- 172.21.15.129:0/2264587668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f1c06c030 0x7f4f1c1f1820 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 >> 172.21.15.129:0/2264587668 conn(0x7f4f1c0cfb20 msgr2=0x7f4f1c0c87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:54.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 shutdown_connections 2024-03-23T15:59:54.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:54.549+0000 7f4f2328f640 1 -- 172.21.15.129:0/2264587668 wait complete. 2024-03-23T15:59:55.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:55.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:56.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:56.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.270+0000 7f5e764f1640 1 Processor -- start 2024-03-23T15:59:56.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.270+0000 7f5e764f1640 1 -- start start 2024-03-23T15:59:56.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.271+0000 7f5e764f1640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5e7015c000 0x7f5e7015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:56.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.271+0000 7f5e764f1640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e7015b290 con 0x7f5e700ceb00 2024-03-23T15:59:56.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.271+0000 7f5e764f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e700ce800 con 0x7f5e700ccd60 2024-03-23T15:59:56.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.271+0000 7f5e764f1640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f5e70154a70 con 0x7f5e7015c000 2024-03-23T15:59:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.272+0000 7f5e74a8e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f5e7015c000 0x7f5e7015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.272+0000 7f5e6ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5e700ccd60 0x7f5e7006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:57502/0 (socket says 172.21.15.129:57502) 2024-03-23T15:59:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.272+0000 7f5e6ffff640 1 -- 172.21.15.129:0/2700217751 learned_addr learned my addr 172.21.15.129:0/2700217751 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:56.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.275+0000 7f5e6effd640 1 -- 172.21.15.129:0/2700217751 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175174558 0 0) 0x7f5e700ce800 con 0x7f5e700ccd60 2024-03-23T15:59:56.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.275+0000 7f5e6effd640 1 -- 172.21.15.129:0/2700217751 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e58003520 con 0x7f5e700ccd60 2024-03-23T15:59:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.279+0000 7f5e74a8e640 1 -- 172.21.15.129:0/2700217751 >> v1:172.21.15.157:6789/0 conn(0x7f5e700ccd60 legacy=0x7f5e7006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.279+0000 7f5e74a8e640 1 -- 172.21.15.129:0/2700217751 >> v1:172.21.15.129:6789/0 conn(0x7f5e700ceb00 legacy=0x7f5e7015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.279+0000 7f5e74a8e640 1 -- 172.21.15.129:0/2700217751 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e700cd950 con 0x7f5e7015c000 2024-03-23T15:59:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.285+0000 7f5e74a8e640 1 --2- 172.21.15.129:0/2700217751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e7015c000 0x7f5e7015ad50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009d90 tx=0x7f5e6802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d27369cbe6abc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:56.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e6effd640 1 -- 172.21.15.129:0/2700217751 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6802cd80 con 0x7f5e7015c000 2024-03-23T15:59:56.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e6effd640 1 -- 172.21.15.129:0/2700217751 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e68034440 con 0x7f5e7015c000 2024-03-23T15:59:56.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e6effd640 1 -- 172.21.15.129:0/2700217751 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e680347d0 con 0x7f5e7015c000 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e764f1640 1 -- 172.21.15.129:0/2700217751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e7015c000 msgr2=0x7f5e7015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e764f1640 1 --2- 172.21.15.129:0/2700217751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e7015c000 0x7f5e7015ad50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009d90 tx=0x7f5e6802bfd0 comp rx=0 tx=0).stop 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e764f1640 1 -- 172.21.15.129:0/2700217751 shutdown_connections 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e764f1640 1 --2- 172.21.15.129:0/2700217751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e7015c000 0x7f5e7015ad50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.291+0000 7f5e764f1640 1 -- 172.21.15.129:0/2700217751 >> 172.21.15.129:0/2700217751 conn(0x7f5e700b1840 msgr2=0x7f5e700b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.292+0000 7f5e764f1640 1 -- 172.21.15.129:0/2700217751 shutdown_connections 2024-03-23T15:59:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.292+0000 7f5e764f1640 1 -- 172.21.15.129:0/2700217751 wait complete. 2024-03-23T15:59:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.293+0000 7f5e764f1640 1 Processor -- start 2024-03-23T15:59:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.293+0000 7f5e764f1640 1 -- start start 2024-03-23T15:59:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e764f1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e700ccd60 0x7f5e7014ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e764f1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e700ceb00 0x7f5e7014b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e764f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e7015c000 0x7f5e701534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e764f1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701e3c90 con 0x7f5e700ceb00 2024-03-23T15:59:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e764f1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e701e5f40 con 0x7f5e700ccd60 2024-03-23T15:59:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e764f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e701540f0 con 0x7f5e7015c000 2024-03-23T15:59:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e74a8e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e7015c000 0x7f5e701534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.294+0000 7f5e6ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e700ccd60 0x7f5e7014ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.295+0000 7f5e74a8e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e7015c000 0x7f5e701534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:36292/0 (socket says 172.21.15.129:36292) 2024-03-23T15:59:56.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.295+0000 7f5e74a8e640 1 -- 172.21.15.129:0/1172603731 learned_addr learned my addr 172.21.15.129:0/1172603731 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:56.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.296+0000 7f5e6f7fe640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e700ceb00 0x7f5e7014b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:56.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.299+0000 7f5e6ffff640 1 -- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e7015c000 msgr2=0x7f5e701534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:56.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.299+0000 7f5e6ffff640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e7015c000 0x7f5e701534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:56.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.299+0000 7f5e6ffff640 1 -- 172.21.15.129:0/1172603731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e700ceb00 msgr2=0x7f5e7014b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:56.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.299+0000 7f5e6ffff640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e700ceb00 0x7f5e7014b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:56.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.299+0000 7f5e6ffff640 1 -- 172.21.15.129:0/1172603731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e68009a70 con 0x7f5e700ccd60 2024-03-23T15:59:56.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.301+0000 7f5e6ffff640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e700ccd60 0x7f5e7014ad90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000c3f0 tx=0x7f5e60001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.307+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60005ce0 con 0x7f5e700ccd60 2024-03-23T15:59:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.307+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e60005e40 con 0x7f5e700ccd60 2024-03-23T15:59:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.307+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e600072d0 con 0x7f5e700ccd60 2024-03-23T15:59:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.307+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e701f0ec0 con 0x7f5e700ccd60 2024-03-23T15:59:56.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.307+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e701e35d0 con 0x7f5e700ccd60 2024-03-23T15:59:56.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.308+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e7004f530 con 0x7f5e700ccd60 2024-03-23T15:59:56.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.660+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e60007430 con 0x7f5e700ccd60 2024-03-23T15:59:56.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.660+0000 7f5e6cff9640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5e5c06b810 0x7f5e5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:56.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.660+0000 7f5e6f7fe640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5e5c06b810 0x7f5e5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:56.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.661+0000 7f5e6f7fe640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5e5c06b810 0x7f5e5c06dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5e7015e570 tx=0x7f5e64009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:56.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.667+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f5e600583a0 con 0x7f5e700ccd60 2024-03-23T15:59:56.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.685+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e60054640 con 0x7f5e700ccd60 2024-03-23T15:59:56.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:56.929+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e700686f0 con 0x7f5e700ccd60 2024-03-23T15:59:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.117+0000 7f5e6cff9640 1 -- 172.21.15.129:0/1172603731 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f5e60017090 con 0x7f5e700ccd60 2024-03-23T15:59:57.120 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:57.120 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:57.120 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:57.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5e5c06b810 msgr2=0x7f5e5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:57.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5e5c06b810 0x7f5e5c06dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5e7015e570 tx=0x7f5e64009040 comp rx=0 tx=0).stop 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e700ccd60 msgr2=0x7f5e7014ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e700ccd60 0x7f5e7014ad90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000c3f0 tx=0x7f5e60001e70 comp rx=0 tx=0).stop 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 shutdown_connections 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e7015c000 0x7f5e701534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e700ceb00 0x7f5e7014b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f5e5c06b810 0x7f5e5c06dcd0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 --2- 172.21.15.129:0/1172603731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e700ccd60 0x7f5e7014ad90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:57.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 >> 172.21.15.129:0/1172603731 conn(0x7f5e700b1840 msgr2=0x7f5e7014d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:57.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 shutdown_connections 2024-03-23T15:59:57.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:57.120+0000 7f5e764f1640 1 -- 172.21.15.129:0/1172603731 wait complete. 2024-03-23T15:59:57.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T15:59:57.739 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T15:59:58.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T15:59:58.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.840+0000 7fe45d8d4640 1 Processor -- start 2024-03-23T15:59:58.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.840+0000 7fe45d8d4640 1 -- start start 2024-03-23T15:59:58.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.841+0000 7fe45d8d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe45815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:58.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.841+0000 7fe45d8d4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe4580c6000 0x7fe4580c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:58.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.841+0000 7fe45d8d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4580cbcb0 con 0x7fe4580caa00 2024-03-23T15:59:58.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.841+0000 7fe45d8d4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fe4580c5be0 con 0x7fe4580c6000 2024-03-23T15:59:58.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.841+0000 7fe45d8d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe4580c69b0 con 0x7fe4581544b0 2024-03-23T15:59:58.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.842+0000 7fe456ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe45815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:58.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.842+0000 7fe4577fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fe4580c6000 0x7fe4580c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:58.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.842+0000 7fe4567fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe4580caa00 0x7fe45806bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:57510/0 (socket says 172.21.15.129:57510) 2024-03-23T15:59:58.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.842+0000 7fe4567fc640 1 -- 172.21.15.129:0/4278008016 learned_addr learned my addr 172.21.15.129:0/4278008016 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T15:59:58.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.845+0000 7fe456ffd640 1 -- 172.21.15.129:0/4278008016 >> v2:172.21.15.129:3300/0 conn(0x7fe4580c6000 msgr2=0x7fe4580c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T15:59:58.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.845+0000 7fe456ffd640 1 --2- 172.21.15.129:0/4278008016 >> v2:172.21.15.129:3300/0 conn(0x7fe4580c6000 0x7fe4580c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:58.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.845+0000 7fe456ffd640 1 -- 172.21.15.129:0/4278008016 >> v1:172.21.15.157:6789/0 conn(0x7fe4580caa00 legacy=0x7fe45806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:58.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.845+0000 7fe456ffd640 1 -- 172.21.15.129:0/4278008016 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45815b280 con 0x7fe4581544b0 2024-03-23T15:59:58.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.845+0000 7fe4577fe640 1 --2- 172.21.15.129:0/4278008016 >> v2:172.21.15.129:3300/0 conn(0x7fe4580c6000 0x7fe4580c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T15:59:58.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.847+0000 7fe456ffd640 1 --2- 172.21.15.129:0/4278008016 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe45815a2e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe4480097a0 tx=0x7fe44802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f349c1d1119b1b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:58.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.852+0000 7fe455ffb640 1 -- 172.21.15.129:0/4278008016 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448030610 con 0x7fe4581544b0 2024-03-23T15:59:58.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.852+0000 7fe455ffb640 1 -- 172.21.15.129:0/4278008016 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe448030770 con 0x7fe4581544b0 2024-03-23T15:59:58.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.852+0000 7fe455ffb640 1 -- 172.21.15.129:0/4278008016 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448030b00 con 0x7fe4581544b0 2024-03-23T15:59:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 -- 172.21.15.129:0/4278008016 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 msgr2=0x7fe45815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/4278008016 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe45815a2e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe4480097a0 tx=0x7fe44802b960 comp rx=0 tx=0).stop 2024-03-23T15:59:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 -- 172.21.15.129:0/4278008016 shutdown_connections 2024-03-23T15:59:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/4278008016 >> v2:172.21.15.129:3300/0 conn(0x7fe4580c6000 0x7fe4580c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/4278008016 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe45815a2e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 -- 172.21.15.129:0/4278008016 >> 172.21.15.129:0/4278008016 conn(0x7fe458061b50 msgr2=0x7fe458061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:58.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 -- 172.21.15.129:0/4278008016 shutdown_connections 2024-03-23T15:59:58.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.853+0000 7fe45d8d4640 1 -- 172.21.15.129:0/4278008016 wait complete. 2024-03-23T15:59:58.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.854+0000 7fe45d8d4640 1 Processor -- start 2024-03-23T15:59:58.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.854+0000 7fe45d8d4640 1 -- start start 2024-03-23T15:59:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe45d8d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4580c6000 0x7fe4580c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe45d8d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4580caa00 0x7fe458141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe45d8d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe458141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe45d8d4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4581f0cc0 con 0x7fe4580caa00 2024-03-23T15:59:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe45d8d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580b4240 con 0x7fe4580c6000 2024-03-23T15:59:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe45d8d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe4581efe10 con 0x7fe4581544b0 2024-03-23T15:59:58.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe4577fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe458141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:58.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.855+0000 7fe4567fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4580caa00 0x7fe458141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:58.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.856+0000 7fe4577fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe458141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:36298/0 (socket says 172.21.15.129:36298) 2024-03-23T15:59:58.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.856+0000 7fe4577fe640 1 -- 172.21.15.129:0/2561797051 learned_addr learned my addr 172.21.15.129:0/2561797051 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T15:59:58.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.856+0000 7fe456ffd640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4580c6000 0x7fe4580c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:58.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.859+0000 7fe4567fc640 1 -- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 msgr2=0x7fe458141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:58.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.859+0000 7fe4567fc640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe458141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:58.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.859+0000 7fe4567fc640 1 -- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4580c6000 msgr2=0x7fe4580c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:58.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.859+0000 7fe4567fc640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4580c6000 0x7fe4580c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:58.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.859+0000 7fe4567fc640 1 -- 172.21.15.129:0/2561797051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe450009a70 con 0x7fe4580caa00 2024-03-23T15:59:58.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.862+0000 7fe4567fc640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4580caa00 0x7fe458141680 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe44c000f60 tx=0x7fe44c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:58.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.864+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c005ce0 con 0x7fe4580caa00 2024-03-23T15:59:58.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.864+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe448009400 con 0x7fe4580caa00 2024-03-23T15:59:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.865+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4581f0390 con 0x7fe4580caa00 2024-03-23T15:59:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.865+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe458158930 con 0x7fe4580caa00 2024-03-23T15:59:58.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.867+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44c005e40 con 0x7fe4580caa00 2024-03-23T15:59:58.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:58.867+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c005230 con 0x7fe4580caa00 2024-03-23T15:59:59.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.122+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe44c005390 con 0x7fe4580caa00 2024-03-23T15:59:59.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.123+0000 7fe43ffff640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe44406b840 0x7fe44406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T15:59:59.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.123+0000 7fe456ffd640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe44406b840 0x7fe44406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T15:59:59.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.124+0000 7fe456ffd640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe44406b840 0x7fe44406dd00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe448036000 tx=0x7fe44803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T15:59:59.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.136+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fe44c058170 con 0x7fe4580caa00 2024-03-23T15:59:59.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.136+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe44c054410 con 0x7fe4580caa00 2024-03-23T15:59:59.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.379+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe458158930 con 0x7fe4580caa00 2024-03-23T15:59:59.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.394+0000 7fe43ffff640 1 -- 172.21.15.129:0/2561797051 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fe458158930 con 0x7fe4580caa00 2024-03-23T15:59:59.397 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T15:59:59.397 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T15:59:59.397 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe44406b840 msgr2=0x7fe44406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe44406b840 0x7fe44406dd00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe448036000 tx=0x7fe44803b040 comp rx=0 tx=0).stop 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4580caa00 msgr2=0x7fe458141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4580caa00 0x7fe458141680 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe44c000f60 tx=0x7fe44c00c800 comp rx=0 tx=0).stop 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 shutdown_connections 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4581544b0 0x7fe458141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4580caa00 0x7fe458141680 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:59.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fe44406b840 0x7fe44406dd00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:59.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 --2- 172.21.15.129:0/2561797051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4580c6000 0x7fe4580c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T15:59:59.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 >> 172.21.15.129:0/2561797051 conn(0x7fe458061b50 msgr2=0x7fe4580cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T15:59:59.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 shutdown_connections 2024-03-23T15:59:59.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T15:59:59.396+0000 7fe45d8d4640 1 -- 172.21.15.129:0/2561797051 wait complete. 2024-03-23T16:00:00.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:00.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:01.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.116+0000 7f8b8d9b5640 1 Processor -- start 2024-03-23T16:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.117+0000 7f8b8d9b5640 1 -- start start 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.117+0000 7f8b8d9b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.117+0000 7f8b8d9b5640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8b880caa00 0x7f8b8806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.117+0000 7f8b8d9b5640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b880cbcb0 con 0x7f8b8806c0b0 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.117+0000 7f8b8d9b5640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f8b880c5be0 con 0x7f8b880caa00 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.117+0000 7f8b8d9b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8815b280 con 0x7f8b881544b0 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.118+0000 7f8b867fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f8b880caa00 0x7f8b8806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.118+0000 7f8b86ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.118+0000 7f8b86ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8815a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:53956/0 (socket says 172.21.15.129:53956) 2024-03-23T16:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.118+0000 7f8b86ffd640 1 -- 172.21.15.129:0/526248870 learned_addr learned my addr 172.21.15.129:0/526248870 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.121+0000 7f8b86ffd640 1 -- 172.21.15.129:0/526248870 >> v2:172.21.15.129:3300/0 conn(0x7f8b880caa00 msgr2=0x7f8b8806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.121+0000 7f8b86ffd640 1 --2- 172.21.15.129:0/526248870 >> v2:172.21.15.129:3300/0 conn(0x7f8b880caa00 0x7f8b8806bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.121+0000 7f8b86ffd640 1 -- 172.21.15.129:0/526248870 >> v1:172.21.15.129:6789/0 conn(0x7f8b8806c0b0 legacy=0x7f8b880c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:01.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.121+0000 7f8b86ffd640 1 -- 172.21.15.129:0/526248870 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8815a9e0 con 0x7f8b881544b0 2024-03-23T16:00:01.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.122+0000 7f8b86ffd640 1 --2- 172.21.15.129:0/526248870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8815a2e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009870 tx=0x7f8b7802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecb0db696b6a2deb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:01.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.126+0000 7f8b85ffb640 1 -- 172.21.15.129:0/526248870 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7802f840 con 0x7f8b881544b0 2024-03-23T16:00:01.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.126+0000 7f8b85ffb640 1 -- 172.21.15.129:0/526248870 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b7802f9a0 con 0x7f8b881544b0 2024-03-23T16:00:01.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.126+0000 7f8b85ffb640 1 -- 172.21.15.129:0/526248870 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7802fd30 con 0x7f8b881544b0 2024-03-23T16:00:01.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/526248870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 msgr2=0x7f8b8815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:01.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/526248870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8815a2e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009870 tx=0x7f8b7802ba20 comp rx=0 tx=0).stop 2024-03-23T16:00:01.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/526248870 shutdown_connections 2024-03-23T16:00:01.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/526248870 >> v2:172.21.15.129:3300/0 conn(0x7f8b880caa00 0x7f8b8806bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/526248870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8815a2e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/526248870 >> 172.21.15.129:0/526248870 conn(0x7f8b88061b50 msgr2=0x7f8b88061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:01.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/526248870 shutdown_connections 2024-03-23T16:00:01.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.127+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/526248870 wait complete. 2024-03-23T16:00:01.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.128+0000 7f8b8d9b5640 1 Processor -- start 2024-03-23T16:00:01.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 -- start start 2024-03-23T16:00:01.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8806c0b0 0x7f8b880cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:01.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880caa00 0x7f8b8813e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:01.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8813ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:01.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880ce8a0 con 0x7f8b880caa00 2024-03-23T16:00:01.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881f4440 con 0x7f8b8806c0b0 2024-03-23T16:00:01.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.129+0000 7f8b8d9b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8b881f3f00 con 0x7f8b881544b0 2024-03-23T16:00:01.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.130+0000 7f8b867fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880caa00 0x7f8b8813e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:01.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.130+0000 7f8b877fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8813ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:01.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.130+0000 7f8b877fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8813ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:53966/0 (socket says 172.21.15.129:53966) 2024-03-23T16:00:01.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.130+0000 7f8b86ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8806c0b0 0x7f8b880cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:01.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.130+0000 7f8b877fe640 1 -- 172.21.15.129:0/3244583633 learned_addr learned my addr 172.21.15.129:0/3244583633 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:01.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.134+0000 7f8b867fc640 1 -- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 msgr2=0x7f8b8813ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:01.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.134+0000 7f8b867fc640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8813ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.134+0000 7f8b867fc640 1 -- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8806c0b0 msgr2=0x7f8b880cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:01.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.134+0000 7f8b867fc640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8806c0b0 0x7f8b880cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.134+0000 7f8b867fc640 1 -- 172.21.15.129:0/3244583633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b780094d0 con 0x7f8b880caa00 2024-03-23T16:00:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.136+0000 7f8b867fc640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880caa00 0x7f8b8813e550 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00e850 tx=0x7f8b7c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.140+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c0095b0 con 0x7f8b880caa00 2024-03-23T16:00:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.140+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b7c00c8e0 con 0x7f8b880caa00 2024-03-23T16:00:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.140+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b881f41a0 con 0x7f8b880caa00 2024-03-23T16:00:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.140+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00cc70 con 0x7f8b880caa00 2024-03-23T16:00:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.140+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b88148360 con 0x7f8b880caa00 2024-03-23T16:00:01.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.141+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b88075740 con 0x7f8b880caa00 2024-03-23T16:00:01.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.164+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b7c013470 con 0x7f8b880caa00 2024-03-23T16:00:01.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.164+0000 7f8b6ffff640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b7406b840 0x7f8b7406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:01.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.165+0000 7f8b86ffd640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b7406b840 0x7f8b7406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:01.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.165+0000 7f8b86ffd640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b7406b840 0x7f8b7406dd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8b78036000 tx=0x7f8b78008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.430+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8b7c014e40 con 0x7f8b880caa00 2024-03-23T16:00:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.430+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b7c055510 con 0x7f8b880caa00 2024-03-23T16:00:01.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.656+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b8814e530 con 0x7f8b880caa00 2024-03-23T16:00:01.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.710+0000 7f8b6ffff640 1 -- 172.21.15.129:0/3244583633 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8b7c021090 con 0x7f8b880caa00 2024-03-23T16:00:01.712 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:01.712 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:01.713 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.712+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b7406b840 msgr2=0x7f8b7406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.712+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b7406b840 0x7f8b7406dd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8b78036000 tx=0x7f8b78008030 comp rx=0 tx=0).stop 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.712+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880caa00 msgr2=0x7f8b8813e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.712+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880caa00 0x7f8b8813e550 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00e850 tx=0x7f8b7c00ed10 comp rx=0 tx=0).stop 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 shutdown_connections 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b881544b0 0x7f8b8813ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880caa00 0x7f8b8813e550 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8b7406b840 0x7f8b7406dd00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 --2- 172.21.15.129:0/3244583633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8806c0b0 0x7f8b880cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:01.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 >> 172.21.15.129:0/3244583633 conn(0x7f8b88061b50 msgr2=0x7f8b8814a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:01.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 shutdown_connections 2024-03-23T16:00:01.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:01.713+0000 7f8b8d9b5640 1 -- 172.21.15.129:0/3244583633 wait complete. 2024-03-23T16:00:02.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:02.330 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:03.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:03.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.432+0000 7f41e148d640 1 Processor -- start 2024-03-23T16:00:03.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.432+0000 7f41e148d640 1 -- start start 2024-03-23T16:00:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41e148d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f41dc1544b0 0x7f41dc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41e148d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc0caa00 0x7f41dc06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41e148d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41dc06c0b0 0x7f41dc0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41e148d640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f41dc0c5be0 con 0x7f41dc1544b0 2024-03-23T16:00:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41e148d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f41dc0cbcb0 con 0x7f41dc06c0b0 2024-03-23T16:00:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41e148d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41dc15b280 con 0x7f41dc0caa00 2024-03-23T16:00:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.433+0000 7f41db7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41dc06c0b0 0x7f41dc0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.434+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc0caa00 0x7f41dc06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.434+0000 7f41db7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41dc06c0b0 0x7f41dc0c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:41070/0 (socket says 172.21.15.129:41070) 2024-03-23T16:00:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.434+0000 7f41db7fe640 1 -- 172.21.15.129:0/423804576 learned_addr learned my addr 172.21.15.129:0/423804576 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.434+0000 7f41daffd640 1 --2- 172.21.15.129:0/423804576 >> v2:172.21.15.129:3300/0 conn(0x7f41dc1544b0 0x7f41dc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.436+0000 7f41db7fe640 1 -- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc0caa00 msgr2=0x7f41dc06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:03.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.436+0000 7f41db7fe640 1 --2- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc0caa00 0x7f41dc06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.436+0000 7f41db7fe640 1 -- 172.21.15.129:0/423804576 >> v2:172.21.15.129:3300/0 conn(0x7f41dc1544b0 msgr2=0x7f41dc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:03.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.436+0000 7f41db7fe640 1 --2- 172.21.15.129:0/423804576 >> v2:172.21.15.129:3300/0 conn(0x7f41dc1544b0 0x7f41dc15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.436+0000 7f41db7fe640 1 -- 172.21.15.129:0/423804576 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc15a9e0 con 0x7f41dc06c0b0 2024-03-23T16:00:03.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.438+0000 7f41db7fe640 1 --2- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc06c0b0 0x7f41dc0c6000 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0c5670 tx=0x7f41d402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15edeaa995dae9a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:03.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.445+0000 7f41da7fc640 1 -- 172.21.15.129:0/423804576 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d402cb00 con 0x7f41dc06c0b0 2024-03-23T16:00:03.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.445+0000 7f41da7fc640 1 -- 172.21.15.129:0/423804576 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d402cc60 con 0x7f41dc06c0b0 2024-03-23T16:00:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.445+0000 7f41da7fc640 1 -- 172.21.15.129:0/423804576 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d40343f0 con 0x7f41dc06c0b0 2024-03-23T16:00:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 -- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc06c0b0 msgr2=0x7f41dc0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 --2- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc06c0b0 0x7f41dc0c6000 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0c5670 tx=0x7f41d402bf60 comp rx=0 tx=0).stop 2024-03-23T16:00:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 -- 172.21.15.129:0/423804576 shutdown_connections 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 --2- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc06c0b0 0x7f41dc0c6000 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 --2- 172.21.15.129:0/423804576 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc0caa00 0x7f41dc06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 --2- 172.21.15.129:0/423804576 >> v2:172.21.15.129:3300/0 conn(0x7f41dc1544b0 0x7f41dc15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 -- 172.21.15.129:0/423804576 >> 172.21.15.129:0/423804576 conn(0x7f41dc061b50 msgr2=0x7f41dc061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 -- 172.21.15.129:0/423804576 shutdown_connections 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.446+0000 7f41e148d640 1 -- 172.21.15.129:0/423804576 wait complete. 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.447+0000 7f41e148d640 1 Processor -- start 2024-03-23T16:00:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 -- start start 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc06c0b0 0x7f41dc149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41dc0caa00 0x7f41dc1403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc1544b0 0x7f41dc140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc149dc0 con 0x7f41dc0caa00 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc0ce8a0 con 0x7f41dc1544b0 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.448+0000 7f41e148d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41dc0cec20 con 0x7f41dc06c0b0 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.449+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41dc0caa00 0x7f41dc1403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.449+0000 7f41daffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc06c0b0 0x7f41dc149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.449+0000 7f41db7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc1544b0 0x7f41dc140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.449+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41dc0caa00 0x7f41dc1403d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43754/0 (socket says 172.21.15.129:43754) 2024-03-23T16:00:03.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.449+0000 7f41d3fff640 1 -- 172.21.15.129:0/1471023734 learned_addr learned my addr 172.21.15.129:0/1471023734 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.453+0000 7f41db7fe640 1 -- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc06c0b0 msgr2=0x7f41dc149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.453+0000 7f41db7fe640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc06c0b0 0x7f41dc149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.453+0000 7f41db7fe640 1 -- 172.21.15.129:0/1471023734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41dc0caa00 msgr2=0x7f41dc1403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.453+0000 7f41db7fe640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41dc0caa00 0x7f41dc1403d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.453+0000 7f41db7fe640 1 -- 172.21.15.129:0/1471023734 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d4009b30 con 0x7f41dc1544b0 2024-03-23T16:00:03.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.455+0000 7f41db7fe640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc1544b0 0x7f41dc140910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41d4032000 tx=0x7f41d4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:03.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.457+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4009510 con 0x7f41dc1544b0 2024-03-23T16:00:03.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.458+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc0cef00 con 0x7f41dc1544b0 2024-03-23T16:00:03.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.458+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc0cf4c0 con 0x7f41dc1544b0 2024-03-23T16:00:03.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.459+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41dc075740 con 0x7f41dc1544b0 2024-03-23T16:00:03.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.679+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d4040470 con 0x7f41dc1544b0 2024-03-23T16:00:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.720+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4040820 con 0x7f41dc1544b0 2024-03-23T16:00:03.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.739+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f41d4040980 con 0x7f41dc1544b0 2024-03-23T16:00:03.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.739+0000 7f41d37fe640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f41bc06b810 0x7f41bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:03.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.740+0000 7f41daffd640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f41bc06b810 0x7f41bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:03.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.740+0000 7f41daffd640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f41bc06b810 0x7f41bc06dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f41cc006fd0 tx=0x7f41cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:03.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.745+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f41d407bc40 con 0x7f41dc1544b0 2024-03-23T16:00:03.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.763+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f41d4077ee0 con 0x7f41dc1544b0 2024-03-23T16:00:03.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:03.987+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41dc157d00 con 0x7f41dc1544b0 2024-03-23T16:00:04.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.007+0000 7f41d37fe640 1 -- 172.21.15.129:0/1471023734 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f41d407b5a0 con 0x7f41dc1544b0 2024-03-23T16:00:04.010 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:04.010 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:04.010 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f41bc06b810 msgr2=0x7f41bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f41bc06b810 0x7f41bc06dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f41cc006fd0 tx=0x7f41cc008040 comp rx=0 tx=0).stop 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc1544b0 msgr2=0x7f41dc140910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc1544b0 0x7f41dc140910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41d4032000 tx=0x7f41d4004a40 comp rx=0 tx=0).stop 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 shutdown_connections 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f41bc06b810 0x7f41bc06dcd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41dc1544b0 0x7f41dc140910 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:04.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41dc0caa00 0x7f41dc1403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:04.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 --2- 172.21.15.129:0/1471023734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41dc06c0b0 0x7f41dc149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:04.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 >> 172.21.15.129:0/1471023734 conn(0x7f41dc061b50 msgr2=0x7f41dc0c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:04.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 shutdown_connections 2024-03-23T16:00:04.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:04.010+0000 7f41e148d640 1 -- 172.21.15.129:0/1471023734 wait complete. 2024-03-23T16:00:04.629 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:04.629 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:05.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.730+0000 7fc5a30a4640 1 Processor -- start 2024-03-23T16:00:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.730+0000 7fc5a30a4640 1 -- start start 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.731+0000 7fc5a30a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.731+0000 7fc5a30a4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc59c0caa00 0x7fc59c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.731+0000 7fc5a30a4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59c0cbcb0 con 0x7fc59c06c0b0 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.731+0000 7fc5a30a4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc59c0c5be0 con 0x7fc59c0caa00 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.731+0000 7fc5a30a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc59c15b280 con 0x7fc59c1544b0 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.732+0000 7fc5a1641640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc59c06c0b0 0x7fc59c0c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:46246/0 (socket says 172.21.15.129:46246) 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.732+0000 7fc5a1641640 1 -- 172.21.15.129:0/2300721110 learned_addr learned my addr 172.21.15.129:0/2300721110 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.732+0000 7fc5a0e40640 1 --2- 172.21.15.129:0/2300721110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:05.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.732+0000 7fc59bfff640 1 --2- 172.21.15.129:0/2300721110 >> v2:172.21.15.129:3300/0 conn(0x7fc59c0caa00 0x7fc59c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:05.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.735+0000 7fc59bfff640 1 -- 172.21.15.129:0/2300721110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 msgr2=0x7fc59c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:05.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.735+0000 7fc59bfff640 1 --2- 172.21.15.129:0/2300721110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:05.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.735+0000 7fc59bfff640 1 -- 172.21.15.129:0/2300721110 >> v1:172.21.15.157:6789/0 conn(0x7fc59c06c0b0 legacy=0x7fc59c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:05.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.735+0000 7fc59bfff640 1 -- 172.21.15.129:0/2300721110 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c15a9e0 con 0x7fc59c0caa00 2024-03-23T16:00:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.737+0000 7fc59bfff640 1 --2- 172.21.15.129:0/2300721110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c0caa00 0x7fc59c06bb70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc5880097a0 tx=0x7fc58802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29643808bfa539b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.740+0000 7fc59b7fe640 1 -- 172.21.15.129:0/2300721110 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588030610 con 0x7fc59c0caa00 2024-03-23T16:00:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.740+0000 7fc59b7fe640 1 -- 172.21.15.129:0/2300721110 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc588030770 con 0x7fc59c0caa00 2024-03-23T16:00:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.740+0000 7fc59b7fe640 1 -- 172.21.15.129:0/2300721110 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588030b00 con 0x7fc59c0caa00 2024-03-23T16:00:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2300721110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c0caa00 msgr2=0x7fc59c06bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2300721110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c0caa00 0x7fc59c06bb70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc5880097a0 tx=0x7fc58802b960 comp rx=0 tx=0).stop 2024-03-23T16:00:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2300721110 shutdown_connections 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2300721110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c0caa00 0x7fc59c06bb70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2300721110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2300721110 >> 172.21.15.129:0/2300721110 conn(0x7fc59c061b50 msgr2=0x7fc59c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2300721110 shutdown_connections 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.741+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2300721110 wait complete. 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.742+0000 7fc5a30a4640 1 Processor -- start 2024-03-23T16:00:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.742+0000 7fc5a30a4640 1 -- start start 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.743+0000 7fc5a30a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 0x7fc59c151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.743+0000 7fc5a30a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59c0caa00 0x7fc59c14a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.743+0000 7fc5a30a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.743+0000 7fc5a30a4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c0ce150 con 0x7fc59c06c0b0 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.743+0000 7fc5a30a4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c149850 con 0x7fc59c0caa00 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.743+0000 7fc5a30a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc59c156ec0 con 0x7fc59c1544b0 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.744+0000 7fc5a0e40640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 0x7fc59c151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.744+0000 7fc5a1641640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.744+0000 7fc59bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59c0caa00 0x7fc59c14a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.744+0000 7fc5a0e40640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 0x7fc59c151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43758/0 (socket says 172.21.15.129:43758) 2024-03-23T16:00:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.744+0000 7fc5a0e40640 1 -- 172.21.15.129:0/2199001722 learned_addr learned my addr 172.21.15.129:0/2199001722 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.747+0000 7fc5a0e40640 1 -- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 msgr2=0x7fc59c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:05.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.747+0000 7fc5a0e40640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:05.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.747+0000 7fc5a0e40640 1 -- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59c0caa00 msgr2=0x7fc59c14a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:05.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.747+0000 7fc5a0e40640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59c0caa00 0x7fc59c14a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:05.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.747+0000 7fc5a0e40640 1 -- 172.21.15.129:0/2199001722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc588009400 con 0x7fc59c06c0b0 2024-03-23T16:00:05.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.749+0000 7fc5a0e40640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 0x7fc59c151520 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc59000e870 tx=0x7fc59000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:05.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.753+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5900094f0 con 0x7fc59c06c0b0 2024-03-23T16:00:05.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.753+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59000c940 con 0x7fc59c06c0b0 2024-03-23T16:00:05.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.753+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c14ae20 con 0x7fc59c06c0b0 2024-03-23T16:00:05.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.753+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59000ccf0 con 0x7fc59c06c0b0 2024-03-23T16:00:05.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.753+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c1568e0 con 0x7fc59c06c0b0 2024-03-23T16:00:05.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.754+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c0b5780 con 0x7fc59c06c0b0 2024-03-23T16:00:05.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.783+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc590013470 con 0x7fc59c06c0b0 2024-03-23T16:00:05.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.784+0000 7fc5997fa640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc58006b890 0x7fc58006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:05.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.784+0000 7fc59bfff640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc58006b890 0x7fc58006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:05.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.785+0000 7fc59bfff640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc58006b890 0x7fc58006dd50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc588036000 tx=0x7fc588008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:05.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:05.789+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc590014e40 con 0x7fc59c06c0b0 2024-03-23T16:00:06.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.041+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc590055480 con 0x7fc59c06c0b0 2024-03-23T16:00:06.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.301+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc59c06c790 con 0x7fc59c06c0b0 2024-03-23T16:00:06.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.314+0000 7fc5997fa640 1 -- 172.21.15.129:0/2199001722 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc590021090 con 0x7fc59c06c0b0 2024-03-23T16:00:06.316 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:06.316 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:06.316 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc58006b890 msgr2=0x7fc58006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc58006b890 0x7fc58006dd50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc588036000 tx=0x7fc588008030 comp rx=0 tx=0).stop 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 msgr2=0x7fc59c151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 0x7fc59c151520 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc59000e870 tx=0x7fc59000ed30 comp rx=0 tx=0).stop 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 shutdown_connections 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc59c1544b0 0x7fc59c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc58006b890 0x7fc58006dd50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:06.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc59c0caa00 0x7fc59c14a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:06.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 --2- 172.21.15.129:0/2199001722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc59c06c0b0 0x7fc59c151520 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:06.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 >> 172.21.15.129:0/2199001722 conn(0x7fc59c061b50 msgr2=0x7fc59c0b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:06.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 shutdown_connections 2024-03-23T16:00:06.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:06.317+0000 7fc5a30a4640 1 -- 172.21.15.129:0/2199001722 wait complete. 2024-03-23T16:00:06.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:06.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:07.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:08.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.035+0000 7f790dc6f640 1 Processor -- start 2024-03-23T16:00:08.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.035+0000 7f790dc6f640 1 -- start start 2024-03-23T16:00:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.036+0000 7f790dc6f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f790806cf70 0x7f79080cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.036+0000 7f790dc6f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f790815df80 0x7f79081540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.036+0000 7f790dc6f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7908070640 con 0x7f790815dba0 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.036+0000 7f790dc6f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f79080c4c00 con 0x7f790806cf70 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.036+0000 7f790dc6f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f790806c240 con 0x7f790815df80 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.037+0000 7f7906ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f790806cf70 0x7f79080cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.037+0000 7f7907fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f790815df80 0x7f79081540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.038+0000 7f7906ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f790806cf70 0x7f79080cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43764/0 (socket says 172.21.15.129:43764) 2024-03-23T16:00:08.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.038+0000 7f7906ffd640 1 -- 172.21.15.129:0/3488836895 learned_addr learned my addr 172.21.15.129:0/3488836895 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:08.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.041+0000 7f7906ffd640 1 -- 172.21.15.129:0/3488836895 >> v2:172.21.15.157:3300/0 conn(0x7f790815df80 msgr2=0x7f79081540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:08.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.041+0000 7f7906ffd640 1 --2- 172.21.15.129:0/3488836895 >> v2:172.21.15.157:3300/0 conn(0x7f790815df80 0x7f79081540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.041+0000 7f7906ffd640 1 -- 172.21.15.129:0/3488836895 >> v1:172.21.15.129:6789/0 conn(0x7f790815dba0 legacy=0x7f790806ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:08.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.041+0000 7f7906ffd640 1 -- 172.21.15.129:0/3488836895 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79080cbb20 con 0x7f790806cf70 2024-03-23T16:00:08.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.041+0000 7f7907fff640 1 --2- 172.21.15.129:0/3488836895 >> v2:172.21.15.157:3300/0 conn(0x7f790815df80 0x7f79081540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T16:00:08.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.045+0000 7f7906ffd640 1 --2- 172.21.15.129:0/3488836895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790806cf70 0x7f79080cca30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009850 tx=0x7f78fc02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f647d2dff1c21be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:08.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.048+0000 7f79067fc640 1 -- 172.21.15.129:0/3488836895 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc004230 con 0x7f790806cf70 2024-03-23T16:00:08.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f79067fc640 1 -- 172.21.15.129:0/3488836895 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78fc004390 con 0x7f790806cf70 2024-03-23T16:00:08.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 -- 172.21.15.129:0/3488836895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790806cf70 msgr2=0x7f79080cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:08.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 --2- 172.21.15.129:0/3488836895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790806cf70 0x7f79080cca30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009850 tx=0x7f78fc02ba10 comp rx=0 tx=0).stop 2024-03-23T16:00:08.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 -- 172.21.15.129:0/3488836895 shutdown_connections 2024-03-23T16:00:08.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 --2- 172.21.15.129:0/3488836895 >> v2:172.21.15.157:3300/0 conn(0x7f790815df80 0x7f79081540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 --2- 172.21.15.129:0/3488836895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790806cf70 0x7f79080cca30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 -- 172.21.15.129:0/3488836895 >> 172.21.15.129:0/3488836895 conn(0x7f7908071300 msgr2=0x7f79080716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 -- 172.21.15.129:0/3488836895 shutdown_connections 2024-03-23T16:00:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.050+0000 7f790dc6f640 1 -- 172.21.15.129:0/3488836895 wait complete. 2024-03-23T16:00:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.052+0000 7f790dc6f640 1 Processor -- start 2024-03-23T16:00:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.052+0000 7f790dc6f640 1 -- start start 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.052+0000 7f790dc6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f790806cf70 0x7f79081e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f790dc6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f790815dba0 0x7f79081598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f790dc6f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790815df80 0x7f79081f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f790dc6f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79081e3d30 con 0x7f790815df80 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f790dc6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7908150f70 con 0x7f790806cf70 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f790dc6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79081efee0 con 0x7f790815dba0 2024-03-23T16:00:08.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f7907fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790815df80 0x7f79081f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:08.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f79077fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f790806cf70 0x7f79081e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:08.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f7906ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f790815dba0 0x7f79081598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:08.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f7907fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790815df80 0x7f79081f0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43768/0 (socket says 172.21.15.129:43768) 2024-03-23T16:00:08.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.053+0000 7f7907fff640 1 -- 172.21.15.129:0/1762516647 learned_addr learned my addr 172.21.15.129:0/1762516647 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.057+0000 7f79077fe640 1 -- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f790815dba0 msgr2=0x7f79081598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.057+0000 7f79077fe640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f790815dba0 0x7f79081598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.057+0000 7f79077fe640 1 -- 172.21.15.129:0/1762516647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790815df80 msgr2=0x7f79081f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.057+0000 7f79077fe640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790815df80 0x7f79081f0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.057+0000 7f79077fe640 1 -- 172.21.15.129:0/1762516647 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7900009a70 con 0x7f790806cf70 2024-03-23T16:00:08.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.060+0000 7f79077fe640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f790806cf70 0x7f79081e5ca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f78f800c370 tx=0x7f78f800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:08.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.064+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f80052d0 con 0x7f790806cf70 2024-03-23T16:00:08.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.064+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f8005450 con 0x7f790806cf70 2024-03-23T16:00:08.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.064+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8007260 con 0x7f790806cf70 2024-03-23T16:00:08.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.064+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78fc009460 con 0x7f790806cf70 2024-03-23T16:00:08.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.064+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79080c7690 con 0x7f790806cf70 2024-03-23T16:00:08.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.065+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79080686f0 con 0x7f790806cf70 2024-03-23T16:00:08.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.089+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f78f80073c0 con 0x7f790806cf70 2024-03-23T16:00:08.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.089+0000 7f78effff640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78f406b810 0x7f78f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:08.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.090+0000 7f7906ffd640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78f406b810 0x7f78f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:08.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.090+0000 7f7906ffd640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78f406b810 0x7f78f406dcd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f78fc035000 tx=0x7f78fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:08.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.340+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f78f8058230 con 0x7f790806cf70 2024-03-23T16:00:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.352+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f78f80544d0 con 0x7f790806cf70 2024-03-23T16:00:08.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.571+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7908075740 con 0x7f790806cf70 2024-03-23T16:00:08.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.592+0000 7f78effff640 1 -- 172.21.15.129:0/1762516647 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f78f8020070 con 0x7f790806cf70 2024-03-23T16:00:08.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:08.595 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:08.595 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78f406b810 msgr2=0x7f78f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78f406b810 0x7f78f406dcd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f78fc035000 tx=0x7f78fc008030 comp rx=0 tx=0).stop 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f790806cf70 msgr2=0x7f79081e5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f790806cf70 0x7f79081e5ca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f78f800c370 tx=0x7f78f800c830 comp rx=0 tx=0).stop 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 shutdown_connections 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f790815df80 0x7f79081f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f790815dba0 0x7f79081598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f78f406b810 0x7f78f406dcd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 --2- 172.21.15.129:0/1762516647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f790806cf70 0x7f79081e5ca0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:08.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 >> 172.21.15.129:0/1762516647 conn(0x7f7908071300 msgr2=0x7f7908158590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:08.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 shutdown_connections 2024-03-23T16:00:08.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:08.595+0000 7f790dc6f640 1 -- 172.21.15.129:0/1762516647 wait complete. 2024-03-23T16:00:09.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:00:09.039 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:00:09.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:09.212 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:10.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.526+0000 7faf9762a640 1 Processor -- start 2024-03-23T16:00:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.526+0000 7faf9762a640 1 -- start start 2024-03-23T16:00:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf9762a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faf9006c030 0x7faf9006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf9762a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf900cbc60 0x7faf9015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf9762a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf9006cae0 con 0x7faf900cc080 2024-03-23T16:00:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf9762a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faf900cabc0 con 0x7faf9006c030 2024-03-23T16:00:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf9762a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf900caea0 con 0x7faf900cbc60 2024-03-23T16:00:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf95bc7640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7faf900cc080 0x7faf9015de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:46776/0 (socket says 172.21.15.129:46776) 2024-03-23T16:00:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf95bc7640 1 -- 172.21.15.129:0/2091437024 learned_addr learned my addr 172.21.15.129:0/2091437024 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.527+0000 7faf94bc5640 1 --2- 172.21.15.129:0/2091437024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf900cbc60 0x7faf9015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.528+0000 7faf953c6640 1 --2- 172.21.15.129:0/2091437024 >> v2:172.21.15.157:3300/0 conn(0x7faf9006c030 0x7faf9006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.530+0000 7faf94bc5640 1 -- 172.21.15.129:0/2091437024 >> v2:172.21.15.157:3300/0 conn(0x7faf9006c030 msgr2=0x7faf9006c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.530+0000 7faf94bc5640 1 --2- 172.21.15.129:0/2091437024 >> v2:172.21.15.157:3300/0 conn(0x7faf9006c030 0x7faf9006c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.530+0000 7faf94bc5640 1 -- 172.21.15.129:0/2091437024 >> v1:172.21.15.129:6789/0 conn(0x7faf900cc080 legacy=0x7faf9015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.530+0000 7faf94bc5640 1 -- 172.21.15.129:0/2091437024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf900cb180 con 0x7faf900cbc60 2024-03-23T16:00:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.532+0000 7faf94bc5640 1 --2- 172.21.15.129:0/2091437024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf900cbc60 0x7faf9015d900 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faf800097a0 tx=0x7faf8002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49d8db32d0db8671 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:10.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.536+0000 7faf87fff640 1 -- 172.21.15.129:0/2091437024 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8002e070 con 0x7faf900cbc60 2024-03-23T16:00:10.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.536+0000 7faf87fff640 1 -- 172.21.15.129:0/2091437024 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf800315a0 con 0x7faf900cbc60 2024-03-23T16:00:10.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.536+0000 7faf87fff640 1 -- 172.21.15.129:0/2091437024 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80031930 con 0x7faf900cbc60 2024-03-23T16:00:10.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.537+0000 7faf9762a640 1 -- 172.21.15.129:0/2091437024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf900cbc60 msgr2=0x7faf9015d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.537+0000 7faf9762a640 1 --2- 172.21.15.129:0/2091437024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf900cbc60 0x7faf9015d900 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faf800097a0 tx=0x7faf8002b840 comp rx=0 tx=0).stop 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.537+0000 7faf9762a640 1 -- 172.21.15.129:0/2091437024 shutdown_connections 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.537+0000 7faf9762a640 1 --2- 172.21.15.129:0/2091437024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf900cbc60 0x7faf9015d900 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.537+0000 7faf9762a640 1 --2- 172.21.15.129:0/2091437024 >> v2:172.21.15.157:3300/0 conn(0x7faf9006c030 0x7faf9006c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.537+0000 7faf9762a640 1 -- 172.21.15.129:0/2091437024 >> 172.21.15.129:0/2091437024 conn(0x7faf900cfb20 msgr2=0x7faf900cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.538+0000 7faf9762a640 1 -- 172.21.15.129:0/2091437024 shutdown_connections 2024-03-23T16:00:10.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.538+0000 7faf9762a640 1 -- 172.21.15.129:0/2091437024 wait complete. 2024-03-23T16:00:10.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.539+0000 7faf9762a640 1 Processor -- start 2024-03-23T16:00:10.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.539+0000 7faf9762a640 1 -- start start 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.540+0000 7faf9762a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 0x7faf9014e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf9762a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf900cbc60 0x7faf901f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf9762a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf900cc080 0x7faf900c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf9762a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faf9015e7b0 con 0x7faf900cc080 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf9762a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf900c6680 con 0x7faf900cbc60 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf9762a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf9015df10 con 0x7faf9006c030 2024-03-23T16:00:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf953c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 0x7faf9014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf95bc7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf900cc080 0x7faf900c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf94bc5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf900cbc60 0x7faf901f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf953c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 0x7faf9014e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50902/0 (socket says 172.21.15.129:50902) 2024-03-23T16:00:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.541+0000 7faf953c6640 1 -- 172.21.15.129:0/3161573505 learned_addr learned my addr 172.21.15.129:0/3161573505 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:10.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.545+0000 7faf953c6640 1 -- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf900cbc60 msgr2=0x7faf901f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:10.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.545+0000 7faf953c6640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf900cbc60 0x7faf901f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:10.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.545+0000 7faf953c6640 1 -- 172.21.15.129:0/3161573505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf900cc080 msgr2=0x7faf900c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:10.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.545+0000 7faf953c6640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf900cc080 0x7faf900c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:10.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.545+0000 7faf953c6640 1 -- 172.21.15.129:0/3161573505 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf80009400 con 0x7faf9006c030 2024-03-23T16:00:10.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.545+0000 7faf95bc7640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf900cc080 0x7faf900c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T16:00:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.548+0000 7faf953c6640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 0x7faf9014e440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00e870 tx=0x7faf8c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.553+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0094f0 con 0x7faf9006c030 2024-03-23T16:00:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.553+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf8c00c940 con 0x7faf9006c030 2024-03-23T16:00:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.553+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c00ccf0 con 0x7faf9006c030 2024-03-23T16:00:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.553+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9015e1f0 con 0x7faf9006c030 2024-03-23T16:00:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.553+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf900c4e80 con 0x7faf9006c030 2024-03-23T16:00:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.554+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf900c4ce0 con 0x7faf9006c030 2024-03-23T16:00:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.818+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faf8c013470 con 0x7faf9006c030 2024-03-23T16:00:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.819+0000 7faf85ffb640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faf7406b810 0x7faf7406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.819+0000 7faf94bc5640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faf7406b810 0x7faf7406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.820+0000 7faf94bc5640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faf7406b810 0x7faf7406dcd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faf80036000 tx=0x7faf8003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.821+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7faf8c014e40 con 0x7faf9006c030 2024-03-23T16:00:10.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:10.832+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faf8c055350 con 0x7faf9006c030 2024-03-23T16:00:11.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.101+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf900cd900 con 0x7faf9006c030 2024-03-23T16:00:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.118+0000 7faf85ffb640 1 -- 172.21.15.129:0/3161573505 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7faf900cd900 con 0x7faf9006c030 2024-03-23T16:00:11.120 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:11.120 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:11.120 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faf7406b810 msgr2=0x7faf7406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faf7406b810 0x7faf7406dcd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7faf80036000 tx=0x7faf8003d040 comp rx=0 tx=0).stop 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 msgr2=0x7faf9014e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 0x7faf9014e440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00e870 tx=0x7faf8c00ed30 comp rx=0 tx=0).stop 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 shutdown_connections 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf900cc080 0x7faf900c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7faf7406b810 0x7faf7406dcd0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf900cbc60 0x7faf901f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 --2- 172.21.15.129:0/3161573505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf9006c030 0x7faf9014e440 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 >> 172.21.15.129:0/3161573505 conn(0x7faf900cfb20 msgr2=0x7faf900ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 shutdown_connections 2024-03-23T16:00:11.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:11.120+0000 7faf9762a640 1 -- 172.21.15.129:0/3161573505 wait complete. 2024-03-23T16:00:11.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:11.738 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:12.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:12.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.843+0000 7ff80a1b6640 1 Processor -- start 2024-03-23T16:00:12.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.844+0000 7ff80a1b6640 1 -- start start 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff80a1b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff80415d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff80a1b6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff80406c830 con 0x7ff80415de40 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff80a1b6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8040cc370 con 0x7ff80406c030 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff80a1b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff80406cae0 con 0x7ff80406c410 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff802ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff80415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff803fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7ff80415de40 0x7ff804164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:46780/0 (socket says 172.21.15.129:46780) 2024-03-23T16:00:12.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.845+0000 7ff803fff640 1 -- 172.21.15.129:0/3014036875 learned_addr learned my addr 172.21.15.129:0/3014036875 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:12.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.848+0000 7ff8027fc640 1 -- 172.21.15.129:0/3014036875 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932552197 0 0) 0x7ff80406c830 con 0x7ff80415de40 2024-03-23T16:00:12.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.848+0000 7ff8027fc640 1 -- 172.21.15.129:0/3014036875 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7ec003520 con 0x7ff80415de40 2024-03-23T16:00:12.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.849+0000 7ff802ffd640 1 -- 172.21.15.129:0/3014036875 >> v1:172.21.15.157:6789/0 conn(0x7ff80406c030 legacy=0x7ff8040cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:12.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.849+0000 7ff802ffd640 1 -- 172.21.15.129:0/3014036875 >> v1:172.21.15.129:6789/0 conn(0x7ff80415de40 legacy=0x7ff804164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:12.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.849+0000 7ff802ffd640 1 -- 172.21.15.129:0/3014036875 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8040cabc0 con 0x7ff80406c410 2024-03-23T16:00:12.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.852+0000 7ff802ffd640 1 --2- 172.21.15.129:0/3014036875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff80415d900 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80097d0 tx=0x7ff7f802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ead71289f0abbcea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:12.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.857+0000 7ff8027fc640 1 -- 172.21.15.129:0/3014036875 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8031890 con 0x7ff80406c410 2024-03-23T16:00:12.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.857+0000 7ff8027fc640 1 -- 172.21.15.129:0/3014036875 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f80319f0 con 0x7ff80406c410 2024-03-23T16:00:12.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.857+0000 7ff8027fc640 1 -- 172.21.15.129:0/3014036875 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8031d80 con 0x7ff80406c410 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3014036875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 msgr2=0x7ff80415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3014036875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff80415d900 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80097d0 tx=0x7ff7f802b930 comp rx=0 tx=0).stop 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3014036875 shutdown_connections 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3014036875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff80415d900 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3014036875 >> 172.21.15.129:0/3014036875 conn(0x7ff8040cfb20 msgr2=0x7ff8040cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3014036875 shutdown_connections 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.858+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3014036875 wait complete. 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.860+0000 7ff80a1b6640 1 Processor -- start 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.860+0000 7ff80a1b6640 1 -- start start 2024-03-23T16:00:12.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.861+0000 7ff80a1b6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff80406c030 0x7ff8040b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.861+0000 7ff80a1b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff8041f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.861+0000 7ff80a1b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80415de40 0x7ff8041543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.861+0000 7ff80a1b6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff8040b4e30 con 0x7ff80415de40 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.861+0000 7ff80a1b6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff8040c3ce0 con 0x7ff80406c030 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.861+0000 7ff80a1b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff8040c3a00 con 0x7ff80406c410 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.862+0000 7ff803fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80415de40 0x7ff8041543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.862+0000 7ff802ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff8041f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.862+0000 7ff8037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff80406c030 0x7ff8040b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.862+0000 7ff803fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80415de40 0x7ff8041543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41710/0 (socket says 172.21.15.129:41710) 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.862+0000 7ff803fff640 1 -- 172.21.15.129:0/3203440006 learned_addr learned my addr 172.21.15.129:0/3203440006 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:12.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.862+0000 7ff802ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff8041f32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50910/0 (socket says 172.21.15.129:50910) 2024-03-23T16:00:12.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.866+0000 7ff8037fe640 1 -- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 msgr2=0x7ff8041f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:12.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.866+0000 7ff8037fe640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff8041f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:12.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.866+0000 7ff8037fe640 1 -- 172.21.15.129:0/3203440006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80415de40 msgr2=0x7ff8041543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:12.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.866+0000 7ff8037fe640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80415de40 0x7ff8041543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:12.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.866+0000 7ff8037fe640 1 -- 172.21.15.129:0/3203440006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f8009400 con 0x7ff80406c030 2024-03-23T16:00:12.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.868+0000 7ff8037fe640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff80406c030 0x7ff8040b53b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c3c0 tx=0x7ff7f400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.873+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40052d0 con 0x7ff80406c030 2024-03-23T16:00:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.873+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f4005430 con 0x7ff80406c030 2024-03-23T16:00:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.873+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8040c64b0 con 0x7ff80406c030 2024-03-23T16:00:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.873+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4003830 con 0x7ff80406c030 2024-03-23T16:00:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.874+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8040b4a60 con 0x7ff80406c030 2024-03-23T16:00:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.875+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8040c69d0 con 0x7ff80406c030 2024-03-23T16:00:12.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.907+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff7f4003990 con 0x7ff80406c030 2024-03-23T16:00:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.908+0000 7ff7ebfff640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff7f006b860 0x7ff7f006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.908+0000 7ff802ffd640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff7f006b860 0x7ff7f006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:12.909+0000 7ff802ffd640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff7f006b860 0x7ff7f006dd20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8036000 tx=0x7ff7f803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.151+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff7f4058030 con 0x7ff80406c030 2024-03-23T16:00:13.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.169+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff7f40542d0 con 0x7ff80406c030 2024-03-23T16:00:13.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.439+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8040c62f0 con 0x7ff80406c030 2024-03-23T16:00:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.453+0000 7ff7ebfff640 1 -- 172.21.15.129:0/3203440006 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7ff7f4017020 con 0x7ff80406c030 2024-03-23T16:00:13.455 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:13.455 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.455+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff7f006b860 msgr2=0x7ff7f006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.455+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff7f006b860 0x7ff7f006dd20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8036000 tx=0x7ff7f803c040 comp rx=0 tx=0).stop 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff80406c030 msgr2=0x7ff8040b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff80406c030 0x7ff8040b53b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c3c0 tx=0x7ff7f400c880 comp rx=0 tx=0).stop 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 shutdown_connections 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80415de40 0x7ff8041543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff80406c410 0x7ff8041f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff7f006b860 0x7ff7f006dd20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 --2- 172.21.15.129:0/3203440006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff80406c030 0x7ff8040b53b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 >> 172.21.15.129:0/3203440006 conn(0x7ff8040cfb20 msgr2=0x7ff8040c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 shutdown_connections 2024-03-23T16:00:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:13.456+0000 7ff80a1b6640 1 -- 172.21.15.129:0/3203440006 wait complete. 2024-03-23T16:00:14.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:14.073 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:15.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:15.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.174+0000 7fc4dd883640 1 Processor -- start 2024-03-23T16:00:15.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.174+0000 7fc4dd883640 1 -- start start 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4dd883640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc4d81544b0 0x7fc4d815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4dd883640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc4d80caa00 0x7fc4d806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4dd883640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d80cbcb0 con 0x7fc4d806c0b0 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4dd883640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d80c5be0 con 0x7fc4d81544b0 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4dd883640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d815b280 con 0x7fc4d80caa00 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4d67fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc4d80caa00 0x7fc4d806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:15.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.176+0000 7fc4d6ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc4d81544b0 0x7fc4d815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:15.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.177+0000 7fc4d67fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc4d80caa00 0x7fc4d806bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:38608/0 (socket says 172.21.15.129:38608) 2024-03-23T16:00:15.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.177+0000 7fc4d67fc640 1 -- 172.21.15.129:0/789907279 learned_addr learned my addr 172.21.15.129:0/789907279 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:15.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.180+0000 7fc4d67fc640 1 -- 172.21.15.129:0/789907279 >> v2:172.21.15.129:3300/0 conn(0x7fc4d81544b0 msgr2=0x7fc4d815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:15.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.180+0000 7fc4d67fc640 1 --2- 172.21.15.129:0/789907279 >> v2:172.21.15.129:3300/0 conn(0x7fc4d81544b0 0x7fc4d815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.180+0000 7fc4d67fc640 1 -- 172.21.15.129:0/789907279 >> v1:172.21.15.129:6789/0 conn(0x7fc4d806c0b0 legacy=0x7fc4d80c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:15.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.180+0000 7fc4d67fc640 1 -- 172.21.15.129:0/789907279 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d815a9e0 con 0x7fc4d80caa00 2024-03-23T16:00:15.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.183+0000 7fc4d67fc640 1 --2- 172.21.15.129:0/789907279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d806bb70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc4d806ce60 tx=0x7fc4cc02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d9197cdccc61498 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:15.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.186+0000 7fc4d5ffb640 1 -- 172.21.15.129:0/789907279 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc030860 con 0x7fc4d80caa00 2024-03-23T16:00:15.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.189+0000 7fc4d5ffb640 1 -- 172.21.15.129:0/789907279 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4cc0309c0 con 0x7fc4d80caa00 2024-03-23T16:00:15.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.189+0000 7fc4d5ffb640 1 -- 172.21.15.129:0/789907279 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc030d50 con 0x7fc4d80caa00 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 -- 172.21.15.129:0/789907279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 msgr2=0x7fc4d806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 --2- 172.21.15.129:0/789907279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d806bb70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc4d806ce60 tx=0x7fc4cc02b9a0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 -- 172.21.15.129:0/789907279 shutdown_connections 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 --2- 172.21.15.129:0/789907279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d806bb70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 --2- 172.21.15.129:0/789907279 >> v2:172.21.15.129:3300/0 conn(0x7fc4d81544b0 0x7fc4d815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 -- 172.21.15.129:0/789907279 >> 172.21.15.129:0/789907279 conn(0x7fc4d8061b50 msgr2=0x7fc4d8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 -- 172.21.15.129:0/789907279 shutdown_connections 2024-03-23T16:00:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.190+0000 7fc4dd883640 1 -- 172.21.15.129:0/789907279 wait complete. 2024-03-23T16:00:15.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.191+0000 7fc4dd883640 1 Processor -- start 2024-03-23T16:00:15.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.191+0000 7fc4dd883640 1 -- start start 2024-03-23T16:00:15.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4dd883640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4d806c0b0 0x7fc4d8155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:15.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4dd883640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d81f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:15.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4dd883640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 0x7fc4d80c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:15.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4dd883640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d81594b0 con 0x7fc4d806c0b0 2024-03-23T16:00:15.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4dd883640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d8154ff0 con 0x7fc4d80caa00 2024-03-23T16:00:15.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4dd883640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d81f0270 con 0x7fc4d81544b0 2024-03-23T16:00:15.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.192+0000 7fc4d6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4d806c0b0 0x7fc4d8155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:15.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.193+0000 7fc4d67fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d81f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:15.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.193+0000 7fc4d77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 0x7fc4d80c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:15.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.193+0000 7fc4d77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 0x7fc4d80c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50920/0 (socket says 172.21.15.129:50920) 2024-03-23T16:00:15.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.193+0000 7fc4d77fe640 1 -- 172.21.15.129:0/2737978075 learned_addr learned my addr 172.21.15.129:0/2737978075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:15.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.197+0000 7fc4d77fe640 1 -- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 msgr2=0x7fc4d81f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:15.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.197+0000 7fc4d77fe640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d81f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.197+0000 7fc4d77fe640 1 -- 172.21.15.129:0/2737978075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4d806c0b0 msgr2=0x7fc4d8155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:15.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.197+0000 7fc4d77fe640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4d806c0b0 0x7fc4d8155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.197+0000 7fc4d77fe640 1 -- 172.21.15.129:0/2737978075 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc0094d0 con 0x7fc4d81544b0 2024-03-23T16:00:15.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.199+0000 7fc4d77fe640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 0x7fc4d80c6270 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c780 tx=0x7fc4d000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.204+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0004b60 con 0x7fc4d81544b0 2024-03-23T16:00:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.204+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d000ce90 con 0x7fc4d81544b0 2024-03-23T16:00:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.204+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d81f4290 con 0x7fc4d81544b0 2024-03-23T16:00:15.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.205+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000a850 con 0x7fc4d81544b0 2024-03-23T16:00:15.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.205+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d81f0530 con 0x7fc4d81544b0 2024-03-23T16:00:15.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.206+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4d8075740 con 0x7fc4d81544b0 2024-03-23T16:00:15.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.239+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc4d0006b50 con 0x7fc4d81544b0 2024-03-23T16:00:15.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.240+0000 7fc4bffff640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc4c406b810 0x7fc4c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:15.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.240+0000 7fc4d6ffd640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc4c406b810 0x7fc4c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:15.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.240+0000 7fc4d6ffd640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc4c406b810 0x7fc4c406dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009aa0 tx=0x7fc4c8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:15.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.527+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc4d0059200 con 0x7fc4d81544b0 2024-03-23T16:00:15.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.546+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc4d00554a0 con 0x7fc4d81544b0 2024-03-23T16:00:15.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.769+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4d814e530 con 0x7fc4d81544b0 2024-03-23T16:00:15.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.783+0000 7fc4bffff640 1 -- 172.21.15.129:0/2737978075 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc4d0058b60 con 0x7fc4d81544b0 2024-03-23T16:00:15.786 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:15.786 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:15.786 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc4c406b810 msgr2=0x7fc4c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc4c406b810 0x7fc4c406dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009aa0 tx=0x7fc4c8009190 comp rx=0 tx=0).stop 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 msgr2=0x7fc4d80c6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 0x7fc4d80c6270 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c780 tx=0x7fc4d000cc40 comp rx=0 tx=0).stop 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 shutdown_connections 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc4d81544b0 0x7fc4d80c6270 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc4c406b810 0x7fc4c406dcd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4d80caa00 0x7fc4d81f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 --2- 172.21.15.129:0/2737978075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4d806c0b0 0x7fc4d8155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:15.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.785+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 >> 172.21.15.129:0/2737978075 conn(0x7fc4d8061b50 msgr2=0x7fc4d8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:15.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.786+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 shutdown_connections 2024-03-23T16:00:15.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:15.786+0000 7fc4dd883640 1 -- 172.21.15.129:0/2737978075 wait complete. 2024-03-23T16:00:16.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:16.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:17.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:17.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.506+0000 7f030f64e640 1 Processor -- start 2024-03-23T16:00:17.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.506+0000 7f030f64e640 1 -- start start 2024-03-23T16:00:17.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030f64e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f030815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:17.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030f64e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f03080ceb00 0x7f030806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:17.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030f64e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030815b290 con 0x7f03080cef40 2024-03-23T16:00:17.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030f64e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f03080ce800 con 0x7f03080ceb00 2024-03-23T16:00:17.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030f64e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0308154a70 con 0x7f03080ccd60 2024-03-23T16:00:17.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030d3ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f030815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:17.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.507+0000 7f030cbe9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f03080ceb00 0x7f030806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:17.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.508+0000 7f030d3ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f030815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50924/0 (socket says 172.21.15.129:50924) 2024-03-23T16:00:17.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.508+0000 7f030dbeb640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f03080cef40 0x7f030815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:46804/0 (socket says 172.21.15.129:46804) 2024-03-23T16:00:17.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.508+0000 7f030d3ea640 1 -- 172.21.15.129:0/1829021203 learned_addr learned my addr 172.21.15.129:0/1829021203 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.510+0000 7f030d3ea640 1 -- 172.21.15.129:0/1829021203 >> v2:172.21.15.129:3300/0 conn(0x7f03080ceb00 msgr2=0x7f030806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.510+0000 7f030d3ea640 1 --2- 172.21.15.129:0/1829021203 >> v2:172.21.15.129:3300/0 conn(0x7f03080ceb00 0x7f030806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.510+0000 7f030d3ea640 1 -- 172.21.15.129:0/1829021203 >> v1:172.21.15.129:6789/0 conn(0x7f03080cef40 legacy=0x7f030815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.510+0000 7f030d3ea640 1 -- 172.21.15.129:0/1829021203 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03080cd950 con 0x7f03080ccd60 2024-03-23T16:00:17.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.512+0000 7f030d3ea640 1 --2- 172.21.15.129:0/1829021203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f030815ad50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0304009870 tx=0x7f030402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9096ac04e5d2ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.516+0000 7f02fffff640 1 -- 172.21.15.129:0/1829021203 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030402f840 con 0x7f03080ccd60 2024-03-23T16:00:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.516+0000 7f02fffff640 1 -- 172.21.15.129:0/1829021203 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030402f9a0 con 0x7f03080ccd60 2024-03-23T16:00:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.516+0000 7f02fffff640 1 -- 172.21.15.129:0/1829021203 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030402fd30 con 0x7f03080ccd60 2024-03-23T16:00:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 -- 172.21.15.129:0/1829021203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 msgr2=0x7f030815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 --2- 172.21.15.129:0/1829021203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f030815ad50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0304009870 tx=0x7f030402ba20 comp rx=0 tx=0).stop 2024-03-23T16:00:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 -- 172.21.15.129:0/1829021203 shutdown_connections 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 --2- 172.21.15.129:0/1829021203 >> v2:172.21.15.129:3300/0 conn(0x7f03080ceb00 0x7f030806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 --2- 172.21.15.129:0/1829021203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f030815ad50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 -- 172.21.15.129:0/1829021203 >> 172.21.15.129:0/1829021203 conn(0x7f03080b1840 msgr2=0x7f03080b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 -- 172.21.15.129:0/1829021203 shutdown_connections 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.517+0000 7f030f64e640 1 -- 172.21.15.129:0/1829021203 wait complete. 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.518+0000 7f030f64e640 1 Processor -- start 2024-03-23T16:00:17.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 -- start start 2024-03-23T16:00:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f03081f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03080ceb00 0x7f030815c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 0x7f0308151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f03081f0240 con 0x7f03080cef40 2024-03-23T16:00:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03081f07c0 con 0x7f03080ceb00 2024-03-23T16:00:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.519+0000 7f030f64e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03081f1190 con 0x7f03080ccd60 2024-03-23T16:00:17.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.520+0000 7f030dbeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 0x7f0308151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:17.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.520+0000 7f030d3ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f03081f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:17.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.520+0000 7f030dbeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 0x7f0308151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41740/0 (socket says 172.21.15.129:41740) 2024-03-23T16:00:17.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.520+0000 7f030dbeb640 1 -- 172.21.15.129:0/1800254994 learned_addr learned my addr 172.21.15.129:0/1800254994 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:17.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.520+0000 7f030cbe9640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03080ceb00 0x7f030815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:17.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.523+0000 7f030dbeb640 1 -- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 msgr2=0x7f03081f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:17.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.523+0000 7f030dbeb640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f03081f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:17.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.523+0000 7f030dbeb640 1 -- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03080ceb00 msgr2=0x7f030815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:17.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.523+0000 7f030dbeb640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03080ceb00 0x7f030815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:17.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.523+0000 7f030dbeb640 1 -- 172.21.15.129:0/1800254994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03040094d0 con 0x7f03080cef40 2024-03-23T16:00:17.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.523+0000 7f030d3ea640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f03081f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T16:00:17.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.526+0000 7f030dbeb640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 0x7f0308151d70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f030000c730 tx=0x7f030000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:17.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.528+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030000ce40 con 0x7f03080cef40 2024-03-23T16:00:17.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.528+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03081571d0 con 0x7f03080cef40 2024-03-23T16:00:17.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.529+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03081f0450 con 0x7f03080cef40 2024-03-23T16:00:17.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.529+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f030804f530 con 0x7f03080cef40 2024-03-23T16:00:17.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.531+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030000a640 con 0x7f03080cef40 2024-03-23T16:00:17.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.531+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030000a9b0 con 0x7f03080cef40 2024-03-23T16:00:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.800+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f030000ab10 con 0x7f03080cef40 2024-03-23T16:00:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.801+0000 7f02fdffb640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f02ec06b810 0x7f02ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.801+0000 7f030d3ea640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f02ec06b810 0x7f02ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:17.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.802+0000 7f030d3ea640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f02ec06b810 0x7f02ec06dcd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0304036000 tx=0x7f0304008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:17.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.816+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0300058410 con 0x7f03080cef40 2024-03-23T16:00:17.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:17.816+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03000546b0 con 0x7f03080cef40 2024-03-23T16:00:18.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.070+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03080686f0 con 0x7f03080cef40 2024-03-23T16:00:18.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.091+0000 7f02fdffb640 1 -- 172.21.15.129:0/1800254994 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f0300021090 con 0x7f03080cef40 2024-03-23T16:00:18.093 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:18.093 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:18.094 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f02ec06b810 msgr2=0x7f02ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f02ec06b810 0x7f02ec06dcd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0304036000 tx=0x7f0304008030 comp rx=0 tx=0).stop 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 msgr2=0x7f0308151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 0x7f0308151d70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f030000c730 tx=0x7f030000cbf0 comp rx=0 tx=0).stop 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 shutdown_connections 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03080cef40 0x7f0308151d70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:18.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f02ec06b810 0x7f02ec06dcd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:18.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03080ceb00 0x7f030815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:18.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 --2- 172.21.15.129:0/1800254994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03080ccd60 0x7f03081f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:18.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 >> 172.21.15.129:0/1800254994 conn(0x7f03080b1840 msgr2=0x7f0308158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:18.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 shutdown_connections 2024-03-23T16:00:18.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:18.094+0000 7f030f64e640 1 -- 172.21.15.129:0/1800254994 wait complete. 2024-03-23T16:00:18.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:18.713 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:19.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:19.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.814+0000 7fc39814f640 1 Processor -- start 2024-03-23T16:00:19.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.814+0000 7fc39814f640 1 -- start start 2024-03-23T16:00:19.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.815+0000 7fc39814f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 0x7fc39015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:19.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.815+0000 7fc39814f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc3900cef40 0x7fc39015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:19.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.815+0000 7fc39814f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39015b290 con 0x7fc3900ccd60 2024-03-23T16:00:19.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.815+0000 7fc39814f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc3900ce800 con 0x7fc3900cef40 2024-03-23T16:00:19.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.815+0000 7fc39814f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc390154a70 con 0x7fc3900ceb00 2024-03-23T16:00:19.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.815+0000 7fc3956ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 0x7fc39015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:19.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.816+0000 7fc3956ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 0x7fc39015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50948/0 (socket says 172.21.15.129:50948) 2024-03-23T16:00:19.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.816+0000 7fc3956ea640 1 -- 172.21.15.129:0/1968712920 learned_addr learned my addr 172.21.15.129:0/1968712920 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.816+0000 7fc3966ec640 1 --2- 172.21.15.129:0/1968712920 >> v2:172.21.15.129:3300/0 conn(0x7fc3900cef40 0x7fc39015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.818+0000 7fc3956ea640 1 -- 172.21.15.129:0/1968712920 >> v2:172.21.15.129:3300/0 conn(0x7fc3900cef40 msgr2=0x7fc39015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:19.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.818+0000 7fc3956ea640 1 --2- 172.21.15.129:0/1968712920 >> v2:172.21.15.129:3300/0 conn(0x7fc3900cef40 0x7fc39015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:19.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.818+0000 7fc3956ea640 1 -- 172.21.15.129:0/1968712920 >> v1:172.21.15.129:6789/0 conn(0x7fc3900ccd60 legacy=0x7fc39006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:19.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.818+0000 7fc3956ea640 1 -- 172.21.15.129:0/1968712920 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3900cd950 con 0x7fc3900ceb00 2024-03-23T16:00:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.820+0000 7fc3956ea640 1 --2- 172.21.15.129:0/1968712920 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 0x7fc39015ad50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc380009870 tx=0x7fc38002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b13b48139107931f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.824+0000 7fc394ee9640 1 -- 172.21.15.129:0/1968712920 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38002f840 con 0x7fc3900ceb00 2024-03-23T16:00:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc394ee9640 1 -- 172.21.15.129:0/1968712920 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38002f9a0 con 0x7fc3900ceb00 2024-03-23T16:00:19.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc394ee9640 1 -- 172.21.15.129:0/1968712920 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38002fd30 con 0x7fc3900ceb00 2024-03-23T16:00:19.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 -- 172.21.15.129:0/1968712920 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 msgr2=0x7fc39015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:19.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 --2- 172.21.15.129:0/1968712920 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 0x7fc39015ad50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc380009870 tx=0x7fc38002ba20 comp rx=0 tx=0).stop 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 -- 172.21.15.129:0/1968712920 shutdown_connections 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 --2- 172.21.15.129:0/1968712920 >> v2:172.21.15.129:3300/0 conn(0x7fc3900cef40 0x7fc39015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 --2- 172.21.15.129:0/1968712920 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ceb00 0x7fc39015ad50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 -- 172.21.15.129:0/1968712920 >> 172.21.15.129:0/1968712920 conn(0x7fc3900b1840 msgr2=0x7fc3900b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 -- 172.21.15.129:0/1968712920 shutdown_connections 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.825+0000 7fc39814f640 1 -- 172.21.15.129:0/1968712920 wait complete. 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 Processor -- start 2024-03-23T16:00:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 -- start start 2024-03-23T16:00:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ccd60 0x7fc3901f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3900ceb00 0x7fc3901f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3900cef40 0x7fc3901521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3901509a0 con 0x7fc3900ceb00 2024-03-23T16:00:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc3901f50d0 con 0x7fc3900cef40 2024-03-23T16:00:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.827+0000 7fc39814f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc390151750 con 0x7fc3900ccd60 2024-03-23T16:00:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.828+0000 7fc395eeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ccd60 0x7fc3901f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.828+0000 7fc3956ea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3900ceb00 0x7fc3901f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.828+0000 7fc395eeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ccd60 0x7fc3901f4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:50950/0 (socket says 172.21.15.129:50950) 2024-03-23T16:00:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.828+0000 7fc395eeb640 1 -- 172.21.15.129:0/1438967499 learned_addr learned my addr 172.21.15.129:0/1438967499 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.828+0000 7fc3966ec640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3900cef40 0x7fc3901521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.831+0000 7fc3956ea640 1 -- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ccd60 msgr2=0x7fc3901f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.831+0000 7fc3956ea640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ccd60 0x7fc3901f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.831+0000 7fc3956ea640 1 -- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3900cef40 msgr2=0x7fc3901521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.831+0000 7fc3956ea640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3900cef40 0x7fc3901521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:19.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.831+0000 7fc3956ea640 1 -- 172.21.15.129:0/1438967499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3800094d0 con 0x7fc3900ceb00 2024-03-23T16:00:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.834+0000 7fc3956ea640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3900ceb00 0x7fc3901f15b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc380033000 tx=0x7fc380004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:19.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.838+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380037bc0 con 0x7fc3900ceb00 2024-03-23T16:00:19.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.838+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc380037d20 con 0x7fc3900ceb00 2024-03-23T16:00:19.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.838+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc390150b90 con 0x7fc3900ceb00 2024-03-23T16:00:19.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.838+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380041640 con 0x7fc3900ceb00 2024-03-23T16:00:19.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.838+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc390151470 con 0x7fc3900ceb00 2024-03-23T16:00:19.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:19.840+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39004f530 con 0x7fc3900ceb00 2024-03-23T16:00:20.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.093+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3800417a0 con 0x7fc3900ceb00 2024-03-23T16:00:20.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.094+0000 7fc3867fc640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc37406b860 0x7fc37406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:20.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.094+0000 7fc395eeb640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc37406b860 0x7fc37406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:20.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.094+0000 7fc395eeb640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc37406b860 0x7fc37406dd20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc38c001840 tx=0x7fc38c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.107+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc38003c070 con 0x7fc3900ceb00 2024-03-23T16:00:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.107+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3800783f0 con 0x7fc3900ceb00 2024-03-23T16:00:20.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.326+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3900686f0 con 0x7fc3900ceb00 2024-03-23T16:00:20.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.339+0000 7fc3867fc640 1 -- 172.21.15.129:0/1438967499 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fc38007bab0 con 0x7fc3900ceb00 2024-03-23T16:00:20.341 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:20.341 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:20.341 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:20.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc37406b860 msgr2=0x7fc37406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:20.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc37406b860 0x7fc37406dd20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc38c001840 tx=0x7fc38c009040 comp rx=0 tx=0).stop 2024-03-23T16:00:20.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3900ceb00 msgr2=0x7fc3901f15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:20.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3900ceb00 0x7fc3901f15b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc380033000 tx=0x7fc380004730 comp rx=0 tx=0).stop 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 shutdown_connections 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc37406b860 0x7fc37406dd20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3900cef40 0x7fc3901521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3900ceb00 0x7fc3901f15b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 --2- 172.21.15.129:0/1438967499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3900ccd60 0x7fc3901f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 >> 172.21.15.129:0/1438967499 conn(0x7fc3900b1840 msgr2=0x7fc390154230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 shutdown_connections 2024-03-23T16:00:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:20.341+0000 7fc39814f640 1 -- 172.21.15.129:0/1438967499 wait complete. 2024-03-23T16:00:20.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:20.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:21.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:22.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.061+0000 7f9c15efb640 1 Processor -- start 2024-03-23T16:00:22.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.061+0000 7f9c15efb640 1 -- start start 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.061+0000 7f9c15efb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 0x7f9c1006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c15efb640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9c100cca30 0x7f9c1006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c15efb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9c1015df80 0x7f9c101540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c15efb640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9c100c4c00 con 0x7f9c100cca30 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c15efb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9c1006c240 con 0x7f9c1015df80 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c15efb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9c100cce70 con 0x7f9c1015dba0 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c0effd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9c100cca30 0x7f9c1006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 0x7f9c1006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.062+0000 7f9c0ffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9c1015df80 0x7f9c101540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.063+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 0x7f9c1006cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49890/0 (socket says 172.21.15.129:49890) 2024-03-23T16:00:22.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.063+0000 7f9c0f7fe640 1 -- 172.21.15.129:0/2147439070 learned_addr learned my addr 172.21.15.129:0/2147439070 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:22.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.065+0000 7f9c0f7fe640 1 -- 172.21.15.129:0/2147439070 >> v2:172.21.15.157:3300/0 conn(0x7f9c1015df80 msgr2=0x7f9c101540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:22.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.065+0000 7f9c0f7fe640 1 --2- 172.21.15.129:0/2147439070 >> v2:172.21.15.157:3300/0 conn(0x7f9c1015df80 0x7f9c101540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.065+0000 7f9c0f7fe640 1 -- 172.21.15.129:0/2147439070 >> v2:172.21.15.129:3300/0 conn(0x7f9c100cca30 msgr2=0x7f9c1006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:22.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.065+0000 7f9c0f7fe640 1 --2- 172.21.15.129:0/2147439070 >> v2:172.21.15.129:3300/0 conn(0x7f9c100cca30 0x7f9c1006ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.065+0000 7f9c0f7fe640 1 -- 172.21.15.129:0/2147439070 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c100cbb20 con 0x7f9c1015dba0 2024-03-23T16:00:22.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.068+0000 7f9c0f7fe640 1 --2- 172.21.15.129:0/2147439070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 0x7f9c1006cf70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009870 tx=0x7f9c0002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476629de6af645a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:22.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.072+0000 7f9c0e7fc640 1 -- 172.21.15.129:0/2147439070 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00030840 con 0x7f9c1015dba0 2024-03-23T16:00:22.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.072+0000 7f9c0e7fc640 1 -- 172.21.15.129:0/2147439070 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c000309a0 con 0x7f9c1015dba0 2024-03-23T16:00:22.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.072+0000 7f9c0e7fc640 1 -- 172.21.15.129:0/2147439070 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00030d30 con 0x7f9c1015dba0 2024-03-23T16:00:22.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 -- 172.21.15.129:0/2147439070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 msgr2=0x7f9c1006cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:22.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 --2- 172.21.15.129:0/2147439070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 0x7f9c1006cf70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009870 tx=0x7f9c0002b980 comp rx=0 tx=0).stop 2024-03-23T16:00:22.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 -- 172.21.15.129:0/2147439070 shutdown_connections 2024-03-23T16:00:22.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 --2- 172.21.15.129:0/2147439070 >> v2:172.21.15.157:3300/0 conn(0x7f9c1015df80 0x7f9c101540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 --2- 172.21.15.129:0/2147439070 >> v2:172.21.15.129:3300/0 conn(0x7f9c100cca30 0x7f9c1006ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 --2- 172.21.15.129:0/2147439070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015dba0 0x7f9c1006cf70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 -- 172.21.15.129:0/2147439070 >> 172.21.15.129:0/2147439070 conn(0x7f9c10071300 msgr2=0x7f9c100716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:22.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 -- 172.21.15.129:0/2147439070 shutdown_connections 2024-03-23T16:00:22.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.073+0000 7f9c15efb640 1 -- 172.21.15.129:0/2147439070 wait complete. 2024-03-23T16:00:22.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.074+0000 7f9c15efb640 1 Processor -- start 2024-03-23T16:00:22.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.074+0000 7f9c15efb640 1 -- start start 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c15efb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c100cca30 0x7f9c10149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c15efb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c1015dba0 0x7f9c10156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c15efb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015df80 0x7f9c101488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c15efb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c100caba0 con 0x7f9c1015dba0 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c15efb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101f0270 con 0x7f9c100cca30 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c15efb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9c101f0550 con 0x7f9c1015df80 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015df80 0x7f9c101488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c0effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c1015dba0 0x7f9c10156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.075+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c100cca30 0x7f9c10149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.076+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015df80 0x7f9c101488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49896/0 (socket says 172.21.15.129:49896) 2024-03-23T16:00:22.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.076+0000 7f9c0ffff640 1 -- 172.21.15.129:0/1089172584 learned_addr learned my addr 172.21.15.129:0/1089172584 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:22.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.079+0000 7f9c0effd640 1 -- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015df80 msgr2=0x7f9c101488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:22.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.079+0000 7f9c0effd640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015df80 0x7f9c101488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.079+0000 7f9c0effd640 1 -- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c100cca30 msgr2=0x7f9c10149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:22.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.079+0000 7f9c0effd640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c100cca30 0x7f9c10149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.079+0000 7f9c0effd640 1 -- 172.21.15.129:0/1089172584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c000094d0 con 0x7f9c1015dba0 2024-03-23T16:00:22.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.081+0000 7f9c0effd640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c1015dba0 0x7f9c10156e00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e840 tx=0x7f9c0400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.085+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c040095a0 con 0x7f9c1015dba0 2024-03-23T16:00:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.085+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0400c940 con 0x7f9c1015dba0 2024-03-23T16:00:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.085+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400ccb0 con 0x7f9c1015dba0 2024-03-23T16:00:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.085+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c101f0830 con 0x7f9c1015dba0 2024-03-23T16:00:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.085+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c101f0df0 con 0x7f9c1015dba0 2024-03-23T16:00:22.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.086+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c10075740 con 0x7f9c1015dba0 2024-03-23T16:00:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.104+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9c04005990 con 0x7f9c1015dba0 2024-03-23T16:00:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.104+0000 7f9bf7fff640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9bfc06b810 0x7f9bfc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.105+0000 7f9c0f7fe640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9bfc06b810 0x7f9bfc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.105+0000 7f9c0f7fe640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9bfc06b810 0x7f9bfc06dcd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c00036000 tx=0x7f9c00008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.108+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9c04058fe0 con 0x7f9c1015dba0 2024-03-23T16:00:22.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.353+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9c04055280 con 0x7f9c1015dba0 2024-03-23T16:00:22.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.606+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c100cdce0 con 0x7f9c1015dba0 2024-03-23T16:00:22.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.627+0000 7f9bf7fff640 1 -- 172.21.15.129:0/1089172584 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9c04021090 con 0x7f9c1015dba0 2024-03-23T16:00:22.630 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:22.630 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:22.631 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:22.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9bfc06b810 msgr2=0x7f9bfc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:22.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9bfc06b810 0x7f9bfc06dcd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c00036000 tx=0x7f9c00008030 comp rx=0 tx=0).stop 2024-03-23T16:00:22.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c1015dba0 msgr2=0x7f9c10156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:22.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c1015dba0 0x7f9c10156e00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e840 tx=0x7f9c0400ed00 comp rx=0 tx=0).stop 2024-03-23T16:00:22.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 shutdown_connections 2024-03-23T16:00:22.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c1015df80 0x7f9c101488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c1015dba0 0x7f9c10156e00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9bfc06b810 0x7f9bfc06dcd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 --2- 172.21.15.129:0/1089172584 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c100cca30 0x7f9c10149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:22.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 >> 172.21.15.129:0/1089172584 conn(0x7f9c10071300 msgr2=0x7f9c100c9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:22.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 shutdown_connections 2024-03-23T16:00:22.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:22.630+0000 7f9c15efb640 1 -- 172.21.15.129:0/1089172584 wait complete. 2024-03-23T16:00:23.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:23.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:24.249 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:24.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.350+0000 7fed20c1e640 1 Processor -- start 2024-03-23T16:00:24.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.350+0000 7fed20c1e640 1 -- start start 2024-03-23T16:00:24.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.351+0000 7fed20c1e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fed1c06cf70 0x7fed1c0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:24.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.351+0000 7fed20c1e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed1c070640 con 0x7fed1c15dba0 2024-03-23T16:00:24.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.351+0000 7fed20c1e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed1c0c4c00 con 0x7fed1c15df80 2024-03-23T16:00:24.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.351+0000 7fed20c1e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fed1c06c240 con 0x7fed1c06cf70 2024-03-23T16:00:24.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.351+0000 7fed19d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fed1c06cf70 0x7fed1c0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:24.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.352+0000 7fed19d9b640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fed1c06cf70 0x7fed1c0cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49830/0 (socket says 172.21.15.129:49830) 2024-03-23T16:00:24.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.352+0000 7fed19d9b640 1 -- 172.21.15.129:0/2643457436 learned_addr learned my addr 172.21.15.129:0/2643457436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.354+0000 7fed19d9b640 1 -- 172.21.15.129:0/2643457436 >> v1:172.21.15.157:6789/0 conn(0x7fed1c15df80 legacy=0x7fed1c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.355+0000 7fed19d9b640 1 -- 172.21.15.129:0/2643457436 >> v1:172.21.15.129:6789/0 conn(0x7fed1c15dba0 legacy=0x7fed1c06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.355+0000 7fed19d9b640 1 -- 172.21.15.129:0/2643457436 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c0cbb20 con 0x7fed1c06cf70 2024-03-23T16:00:24.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.358+0000 7fed19d9b640 1 --2- 172.21.15.129:0/2643457436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c0cca30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fed10009880 tx=0x7fed1002baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17c7a6f60a5ea83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.362+0000 7fed1959a640 1 -- 172.21.15.129:0/2643457436 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10004230 con 0x7fed1c06cf70 2024-03-23T16:00:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.362+0000 7fed1959a640 1 -- 172.21.15.129:0/2643457436 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed10004390 con 0x7fed1c06cf70 2024-03-23T16:00:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.362+0000 7fed1959a640 1 -- 172.21.15.129:0/2643457436 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1002d860 con 0x7fed1c06cf70 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 -- 172.21.15.129:0/2643457436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 msgr2=0x7fed1c0cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2643457436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c0cca30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fed10009880 tx=0x7fed1002baa0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 -- 172.21.15.129:0/2643457436 shutdown_connections 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2643457436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c0cca30 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 -- 172.21.15.129:0/2643457436 >> 172.21.15.129:0/2643457436 conn(0x7fed1c071300 msgr2=0x7fed1c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 -- 172.21.15.129:0/2643457436 shutdown_connections 2024-03-23T16:00:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.363+0000 7fed20c1e640 1 -- 172.21.15.129:0/2643457436 wait complete. 2024-03-23T16:00:24.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.365+0000 7fed20c1e640 1 Processor -- start 2024-03-23T16:00:24.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.365+0000 7fed20c1e640 1 -- start start 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.366+0000 7fed20c1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c1e5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.366+0000 7fed20c1e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed1c15dba0 0x7fed1c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.366+0000 7fed20c1e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed1c15df80 0x7fed1c158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.366+0000 7fed20c1e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c150e40 con 0x7fed1c06cf70 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.366+0000 7fed20c1e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c158120 con 0x7fed1c15df80 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.366+0000 7fed20c1e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c150310 con 0x7fed1c15dba0 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.367+0000 7fed1a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c1e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.367+0000 7fed19d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed1c15dba0 0x7fed1c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:24.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.367+0000 7fed1ad9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed1c15df80 0x7fed1c158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:24.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.367+0000 7fed1a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c1e5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49846/0 (socket says 172.21.15.129:49846) 2024-03-23T16:00:24.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.367+0000 7fed1a59c640 1 -- 172.21.15.129:0/2799366478 learned_addr learned my addr 172.21.15.129:0/2799366478 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:24.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.371+0000 7fed1a59c640 1 -- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed1c15dba0 msgr2=0x7fed1c1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.371+0000 7fed1a59c640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed1c15dba0 0x7fed1c1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.371+0000 7fed1a59c640 1 -- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed1c15df80 msgr2=0x7fed1c158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.371+0000 7fed1a59c640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed1c15df80 0x7fed1c158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.371+0000 7fed1a59c640 1 -- 172.21.15.129:0/2799366478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed10009460 con 0x7fed1c06cf70 2024-03-23T16:00:24.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.371+0000 7fed19d9b640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed1c15dba0 0x7fed1c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T16:00:24.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.373+0000 7fed1a59c640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c1e5560 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c430 tx=0x7fed0c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:24.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.377+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0052d0 con 0x7fed1c06cf70 2024-03-23T16:00:24.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.377+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c005450 con 0x7fed1c06cf70 2024-03-23T16:00:24.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.377+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c007260 con 0x7fed1c06cf70 2024-03-23T16:00:24.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.377+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c1505a0 con 0x7fed1c06cf70 2024-03-23T16:00:24.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.378+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c150b60 con 0x7fed1c06cf70 2024-03-23T16:00:24.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.379+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1c075740 con 0x7fed1c06cf70 2024-03-23T16:00:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.402+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fed0c0073c0 con 0x7fed1c06cf70 2024-03-23T16:00:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.403+0000 7fed0affd640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fecfc06b860 0x7fecfc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.403+0000 7fed19d9b640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fecfc06b860 0x7fecfc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.403+0000 7fed19d9b640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fecfc06b860 0x7fecfc06dd20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fed10035000 tx=0x7fed10008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:24.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.407+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fed0c007670 con 0x7fed1c06cf70 2024-03-23T16:00:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.687+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fed0c054550 con 0x7fed1c06cf70 2024-03-23T16:00:24.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.954+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed1c0cd310 con 0x7fed1c06cf70 2024-03-23T16:00:24.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.975+0000 7fed0affd640 1 -- 172.21.15.129:0/2799366478 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fed0c057c10 con 0x7fed1c06cf70 2024-03-23T16:00:24.978 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:24.978 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:24.978 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:24.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fecfc06b860 msgr2=0x7fecfc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fecfc06b860 0x7fecfc06dd20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fed10035000 tx=0x7fed10008030 comp rx=0 tx=0).stop 2024-03-23T16:00:24.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 msgr2=0x7fed1c1e5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:24.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c1e5560 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c430 tx=0x7fed0c00c8f0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 shutdown_connections 2024-03-23T16:00:24.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fecfc06b860 0x7fecfc06dd20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed1c15df80 0x7fed1c158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed1c15dba0 0x7fed1c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 --2- 172.21.15.129:0/2799366478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed1c06cf70 0x7fed1c1e5560 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:24.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 >> 172.21.15.129:0/2799366478 conn(0x7fed1c071300 msgr2=0x7fed1c0c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:24.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 shutdown_connections 2024-03-23T16:00:24.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:24.978+0000 7fed20c1e640 1 -- 172.21.15.129:0/2799366478 wait complete. 2024-03-23T16:00:25.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:25.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:26.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:26.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.704+0000 7f8978c74640 1 Processor -- start 2024-03-23T16:00:26.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.704+0000 7f8978c74640 1 -- start start 2024-03-23T16:00:26.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.705+0000 7f8978c74640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f897415d9c0 0x7f897406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:26.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.705+0000 7f8978c74640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f89740ca4a0 0x7f89740cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.705+0000 7f8978c74640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89740cb810 con 0x7f897415dda0 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.705+0000 7f8978c74640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f89740cac00 con 0x7f897415d9c0 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.705+0000 7f8978c74640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f89740caee0 con 0x7f89740ca4a0 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.706+0000 7f897259c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f897415d9c0 0x7f897406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.706+0000 7f896bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f89740ca4a0 0x7f89740cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.706+0000 7f896bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f89740ca4a0 0x7f89740cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:57870/0 (socket says 172.21.15.129:57870) 2024-03-23T16:00:26.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.706+0000 7f896bfff640 1 -- 172.21.15.129:0/1746719140 learned_addr learned my addr 172.21.15.129:0/1746719140 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:26.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.708+0000 7f896bfff640 1 -- 172.21.15.129:0/1746719140 >> v2:172.21.15.129:3300/0 conn(0x7f897415d9c0 msgr2=0x7f897406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:26.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.708+0000 7f896bfff640 1 --2- 172.21.15.129:0/1746719140 >> v2:172.21.15.129:3300/0 conn(0x7f897415d9c0 0x7f897406c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:26.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.708+0000 7f896bfff640 1 -- 172.21.15.129:0/1746719140 >> v1:172.21.15.129:6789/0 conn(0x7f897415dda0 legacy=0x7f897415e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:26.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.709+0000 7f896bfff640 1 -- 172.21.15.129:0/1746719140 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89740cb1c0 con 0x7f89740ca4a0 2024-03-23T16:00:26.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.711+0000 7f896bfff640 1 --2- 172.21.15.129:0/1746719140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89740ca4a0 0x7f89740cbca0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f897408ced0 tx=0x7f895c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f64d7521d558109c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:26.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.715+0000 7f8971d9b640 1 -- 172.21.15.129:0/1746719140 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c030860 con 0x7f89740ca4a0 2024-03-23T16:00:26.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.715+0000 7f8971d9b640 1 -- 172.21.15.129:0/1746719140 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f895c0309c0 con 0x7f89740ca4a0 2024-03-23T16:00:26.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.715+0000 7f8971d9b640 1 -- 172.21.15.129:0/1746719140 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c030d50 con 0x7f89740ca4a0 2024-03-23T16:00:26.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 -- 172.21.15.129:0/1746719140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89740ca4a0 msgr2=0x7f89740cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:26.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 --2- 172.21.15.129:0/1746719140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89740ca4a0 0x7f89740cbca0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f897408ced0 tx=0x7f895c02b9a0 comp rx=0 tx=0).stop 2024-03-23T16:00:26.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 -- 172.21.15.129:0/1746719140 shutdown_connections 2024-03-23T16:00:26.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 --2- 172.21.15.129:0/1746719140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89740ca4a0 0x7f89740cbca0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:26.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 --2- 172.21.15.129:0/1746719140 >> v2:172.21.15.129:3300/0 conn(0x7f897415d9c0 0x7f897406c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:26.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 -- 172.21.15.129:0/1746719140 >> 172.21.15.129:0/1746719140 conn(0x7f89740cfb20 msgr2=0x7f89740cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:26.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 -- 172.21.15.129:0/1746719140 shutdown_connections 2024-03-23T16:00:26.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.716+0000 7f8978c74640 1 -- 172.21.15.129:0/1746719140 wait complete. 2024-03-23T16:00:26.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.718+0000 7f8978c74640 1 Processor -- start 2024-03-23T16:00:26.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.718+0000 7f8978c74640 1 -- start start 2024-03-23T16:00:26.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.719+0000 7f8978c74640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89740ca4a0 0x7f8974158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:26.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.719+0000 7f8978c74640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897415d9c0 0x7f89741e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:26.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.719+0000 7f8978c74640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 0x7f8974153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:26.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.719+0000 7f8978c74640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89741580b0 con 0x7f897415dda0 2024-03-23T16:00:26.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.719+0000 7f8978c74640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f897415a080 con 0x7f897415d9c0 2024-03-23T16:00:26.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.719+0000 7f8978c74640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8974158a40 con 0x7f89740ca4a0 2024-03-23T16:00:26.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.720+0000 7f896bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897415d9c0 0x7f89741e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:26.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.720+0000 7f8972d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 0x7f8974153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:26.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.720+0000 7f897259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89740ca4a0 0x7f8974158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:26.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.720+0000 7f8972d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 0x7f8974153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49868/0 (socket says 172.21.15.129:49868) 2024-03-23T16:00:26.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.720+0000 7f8972d9d640 1 -- 172.21.15.129:0/2737384740 learned_addr learned my addr 172.21.15.129:0/2737384740 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:26.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.724+0000 7f8972d9d640 1 -- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89740ca4a0 msgr2=0x7f8974158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:26.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.724+0000 7f8972d9d640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89740ca4a0 0x7f8974158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:26.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.724+0000 7f8972d9d640 1 -- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897415d9c0 msgr2=0x7f89741e5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:26.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.724+0000 7f8972d9d640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897415d9c0 0x7f89741e5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:26.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.724+0000 7f8972d9d640 1 -- 172.21.15.129:0/2737384740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c0094d0 con 0x7f897415dda0 2024-03-23T16:00:26.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.727+0000 7f8972d9d640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 0x7f8974153c70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f896c00c730 tx=0x7f896c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.731+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c003b70 con 0x7f897415dda0 2024-03-23T16:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.731+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f896c003cd0 con 0x7f897415dda0 2024-03-23T16:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.731+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c00aa90 con 0x7f897415dda0 2024-03-23T16:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.731+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89740b5740 con 0x7f897415dda0 2024-03-23T16:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.731+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897415a5d0 con 0x7f897415dda0 2024-03-23T16:00:26.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.732+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8974158220 con 0x7f897415dda0 2024-03-23T16:00:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.989+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f896c00abf0 con 0x7f897415dda0 2024-03-23T16:00:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.990+0000 7f896affd640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f895406b810 0x7f895406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.990+0000 7f897259c640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f895406b810 0x7f895406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:26.991+0000 7f897259c640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f895406b810 0x7f895406dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8964009aa0 tx=0x7f8964009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:27.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.003+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f896c00ce40 con 0x7f897415dda0 2024-03-23T16:00:27.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.003+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f896c0552c0 con 0x7f897415dda0 2024-03-23T16:00:27.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.278+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89741581f0 con 0x7f897415dda0 2024-03-23T16:00:27.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.294+0000 7f896affd640 1 -- 172.21.15.129:0/2737384740 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f896c058980 con 0x7f897415dda0 2024-03-23T16:00:27.297 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:27.297 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:27.297 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f895406b810 msgr2=0x7f895406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f895406b810 0x7f895406dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8964009aa0 tx=0x7f8964009190 comp rx=0 tx=0).stop 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 msgr2=0x7f8974153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 0x7f8974153c70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f896c00c730 tx=0x7f896c00cbf0 comp rx=0 tx=0).stop 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 shutdown_connections 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f897415dda0 0x7f8974153c70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f895406b810 0x7f895406dcd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897415d9c0 0x7f89741e5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:27.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 --2- 172.21.15.129:0/2737384740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89740ca4a0 0x7f8974158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:27.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 >> 172.21.15.129:0/2737384740 conn(0x7f89740cfb20 msgr2=0x7f897415c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:27.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 shutdown_connections 2024-03-23T16:00:27.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:27.297+0000 7f8978c74640 1 -- 172.21.15.129:0/2737384740 wait complete. 2024-03-23T16:00:27.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:27.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:28.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:29.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.015+0000 7f718ef3e640 1 Processor -- start 2024-03-23T16:00:29.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.015+0000 7f718ef3e640 1 -- start start 2024-03-23T16:00:29.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.016+0000 7f718ef3e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7188154340 0x7f71880c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:29.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.016+0000 7f718ef3e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f718806bd40 0x7f71880cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:29.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.016+0000 7f718ef3e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f718815e090 con 0x7f718815e520 2024-03-23T16:00:29.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.016+0000 7f718ef3e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f718815d860 con 0x7f718806bd40 2024-03-23T16:00:29.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.016+0000 7f718ef3e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f718806ca10 con 0x7f7188154340 2024-03-23T16:00:29.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.017+0000 7f718d4db640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f718806bd40 0x7f71880cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:29.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.017+0000 7f718ccda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7188154340 0x7f71880c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:29.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.017+0000 7f718d4db640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f718806bd40 0x7f71880cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:57884/0 (socket says 172.21.15.129:57884) 2024-03-23T16:00:29.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.017+0000 7f718d4db640 1 -- 172.21.15.129:0/251376104 learned_addr learned my addr 172.21.15.129:0/251376104 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.020+0000 7f718d4db640 1 -- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7188154340 msgr2=0x7f71880c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.020+0000 7f718d4db640 1 --2- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7188154340 0x7f71880c6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.020+0000 7f718d4db640 1 -- 172.21.15.129:0/251376104 >> v1:172.21.15.157:6789/0 conn(0x7f718815e520 legacy=0x7f718815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.020+0000 7f718d4db640 1 -- 172.21.15.129:0/251376104 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71880c74d0 con 0x7f718806bd40 2024-03-23T16:00:29.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.026+0000 7f718d4db640 1 --2- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f71880cc160 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f718002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb75332f8915dde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:29.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.032+0000 7f71877fe640 1 -- 172.21.15.129:0/251376104 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718002cb90 con 0x7f718806bd40 2024-03-23T16:00:29.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.032+0000 7f71877fe640 1 -- 172.21.15.129:0/251376104 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f718002ccf0 con 0x7f718806bd40 2024-03-23T16:00:29.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.032+0000 7f71877fe640 1 -- 172.21.15.129:0/251376104 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180034610 con 0x7f718806bd40 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 -- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 msgr2=0x7f71880cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 --2- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f71880cc160 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f718002bff0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 -- 172.21.15.129:0/251376104 shutdown_connections 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 --2- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f71880cc160 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 --2- 172.21.15.129:0/251376104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7188154340 0x7f71880c6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 -- 172.21.15.129:0/251376104 >> 172.21.15.129:0/251376104 conn(0x7f7188071dc0 msgr2=0x7f71880721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 -- 172.21.15.129:0/251376104 shutdown_connections 2024-03-23T16:00:29.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.033+0000 7f718ef3e640 1 -- 172.21.15.129:0/251376104 wait complete. 2024-03-23T16:00:29.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.034+0000 7f718ef3e640 1 Processor -- start 2024-03-23T16:00:29.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.034+0000 7f718ef3e640 1 -- start start 2024-03-23T16:00:29.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ef3e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f718814bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:29.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ef3e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7188154340 0x7f718814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:29.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ef3e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 0x7f7188150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:29.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ef3e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71881584b0 con 0x7f7188154340 2024-03-23T16:00:29.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ef3e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f718814f200 con 0x7f718806bd40 2024-03-23T16:00:29.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ef3e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f718814ef60 con 0x7f718815e520 2024-03-23T16:00:29.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f7187fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7188154340 0x7f718814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:29.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718ccda640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f718814bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:29.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.035+0000 7f718d4db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 0x7f7188150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:29.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.036+0000 7f718d4db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 0x7f7188150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:49928/0 (socket says 172.21.15.129:49928) 2024-03-23T16:00:29.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.036+0000 7f718d4db640 1 -- 172.21.15.129:0/2646092838 learned_addr learned my addr 172.21.15.129:0/2646092838 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:29.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.041+0000 7f718d4db640 1 -- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 msgr2=0x7f718814bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:29.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.041+0000 7f718d4db640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f718814bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.041+0000 7f718d4db640 1 -- 172.21.15.129:0/2646092838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7188154340 msgr2=0x7f718814fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:29.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.041+0000 7f718d4db640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7188154340 0x7f718814fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.041+0000 7f718d4db640 1 -- 172.21.15.129:0/2646092838 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7180009b30 con 0x7f718815e520 2024-03-23T16:00:29.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.044+0000 7f718d4db640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 0x7f7188150150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f7180004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:29.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.048+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180009510 con 0x7f718815e520 2024-03-23T16:00:29.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.048+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f718003e450 con 0x7f718815e520 2024-03-23T16:00:29.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.048+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718003e7e0 con 0x7f718815e520 2024-03-23T16:00:29.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.048+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71881f04a0 con 0x7f718815e520 2024-03-23T16:00:29.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.048+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71881f0740 con 0x7f718815e520 2024-03-23T16:00:29.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.049+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7188075740 con 0x7f718815e520 2024-03-23T16:00:29.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.069+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f718003e940 con 0x7f718815e520 2024-03-23T16:00:29.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.069+0000 7f71857fa640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f716c06b860 0x7f716c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:29.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.070+0000 7f718ccda640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f716c06b860 0x7f716c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:29.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.070+0000 7f718ccda640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f716c06b860 0x7f716c06dd20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f717c009b40 tx=0x7f717c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:29.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.311+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f718007b7b0 con 0x7f718815e520 2024-03-23T16:00:29.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.322+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7180077a50 con 0x7f718815e520 2024-03-23T16:00:29.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.540+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f718806bb20 con 0x7f718815e520 2024-03-23T16:00:29.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.612+0000 7f71857fa640 1 -- 172.21.15.129:0/2646092838 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f7180043090 con 0x7f718815e520 2024-03-23T16:00:29.615 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:29.615 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:29.615 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f716c06b860 msgr2=0x7f716c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f716c06b860 0x7f716c06dd20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f717c009b40 tx=0x7f717c009190 comp rx=0 tx=0).stop 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 msgr2=0x7f7188150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 0x7f7188150150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f7180004a40 comp rx=0 tx=0).stop 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 shutdown_connections 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f718815e520 0x7f7188150150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7188154340 0x7f718814fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f716c06b860 0x7f716c06dd20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 --2- 172.21.15.129:0/2646092838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f718806bd40 0x7f718814bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 >> 172.21.15.129:0/2646092838 conn(0x7f7188071dc0 msgr2=0x7f71881522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:29.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 shutdown_connections 2024-03-23T16:00:29.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:29.615+0000 7f718ef3e640 1 -- 172.21.15.129:0/2646092838 wait complete. 2024-03-23T16:00:30.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:30.232 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:31.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:31.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.334+0000 7fae6140e640 1 Processor -- start 2024-03-23T16:00:31.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.334+0000 7fae6140e640 1 -- start start 2024-03-23T16:00:31.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae6140e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fae5c15a2e0 0x7fae5c06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:31.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae6140e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fae5c06c0b0 0x7fae5c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:31.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae6140e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae5c0cbcb0 con 0x7fae5c1544b0 2024-03-23T16:00:31.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae6140e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fae5c0c5be0 con 0x7fae5c06c0b0 2024-03-23T16:00:31.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae6140e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fae5c15b280 con 0x7fae5c15a2e0 2024-03-23T16:00:31.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae5b7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fae5c06c0b0 0x7fae5c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:31.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.335+0000 7fae5a7fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fae5c15a2e0 0x7fae5c06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:31.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.336+0000 7fae5affd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fae5c1544b0 0x7fae5c0caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:52246/0 (socket says 172.21.15.129:52246) 2024-03-23T16:00:31.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.336+0000 7fae5affd640 1 -- 172.21.15.129:0/918000812 learned_addr learned my addr 172.21.15.129:0/918000812 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:31.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.339+0000 7fae5b7fe640 1 -- 172.21.15.129:0/918000812 >> v2:172.21.15.157:3300/0 conn(0x7fae5c15a2e0 msgr2=0x7fae5c06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:31.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.339+0000 7fae5b7fe640 1 --2- 172.21.15.129:0/918000812 >> v2:172.21.15.157:3300/0 conn(0x7fae5c15a2e0 0x7fae5c06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.339+0000 7fae5b7fe640 1 -- 172.21.15.129:0/918000812 >> v1:172.21.15.129:6789/0 conn(0x7fae5c1544b0 legacy=0x7fae5c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:31.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.339+0000 7fae5b7fe640 1 -- 172.21.15.129:0/918000812 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c15a9e0 con 0x7fae5c06c0b0 2024-03-23T16:00:31.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.342+0000 7fae5b7fe640 1 --2- 172.21.15.129:0/918000812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0c6000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fae5c06ce60 tx=0x7fae5402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36861a6c6249e417 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:31.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.346+0000 7fae59ffb640 1 -- 172.21.15.129:0/918000812 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5402cd00 con 0x7fae5c06c0b0 2024-03-23T16:00:31.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.346+0000 7fae59ffb640 1 -- 172.21.15.129:0/918000812 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5402ce60 con 0x7fae5c06c0b0 2024-03-23T16:00:31.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.346+0000 7fae59ffb640 1 -- 172.21.15.129:0/918000812 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54034660 con 0x7fae5c06c0b0 2024-03-23T16:00:31.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 -- 172.21.15.129:0/918000812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c06c0b0 msgr2=0x7fae5c0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:31.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 --2- 172.21.15.129:0/918000812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0c6000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fae5c06ce60 tx=0x7fae5402bf50 comp rx=0 tx=0).stop 2024-03-23T16:00:31.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 -- 172.21.15.129:0/918000812 shutdown_connections 2024-03-23T16:00:31.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 --2- 172.21.15.129:0/918000812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0c6000 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 --2- 172.21.15.129:0/918000812 >> v2:172.21.15.157:3300/0 conn(0x7fae5c15a2e0 0x7fae5c06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 -- 172.21.15.129:0/918000812 >> 172.21.15.129:0/918000812 conn(0x7fae5c061b50 msgr2=0x7fae5c061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:31.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 -- 172.21.15.129:0/918000812 shutdown_connections 2024-03-23T16:00:31.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.347+0000 7fae6140e640 1 -- 172.21.15.129:0/918000812 wait complete. 2024-03-23T16:00:31.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.349+0000 7fae6140e640 1 Processor -- start 2024-03-23T16:00:31.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.349+0000 7fae6140e640 1 -- start start 2024-03-23T16:00:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.350+0000 7fae6140e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.350+0000 7fae6140e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae5c1544b0 0x7fae5c145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.350+0000 7fae6140e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 0x7fae5c0cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.350+0000 7fae6140e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c144490 con 0x7fae5c15a2e0 2024-03-23T16:00:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.350+0000 7fae6140e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c0d05a0 con 0x7fae5c06c0b0 2024-03-23T16:00:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.350+0000 7fae6140e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c0cfc30 con 0x7fae5c1544b0 2024-03-23T16:00:31.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.351+0000 7fae5b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 0x7fae5c0cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:31.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.351+0000 7fae5affd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:31.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.351+0000 7fae5a7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae5c1544b0 0x7fae5c145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:31.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.351+0000 7fae5b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 0x7fae5c0cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43622/0 (socket says 172.21.15.129:43622) 2024-03-23T16:00:31.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.351+0000 7fae5b7fe640 1 -- 172.21.15.129:0/1606505933 learned_addr learned my addr 172.21.15.129:0/1606505933 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.354+0000 7fae5b7fe640 1 -- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae5c1544b0 msgr2=0x7fae5c145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.354+0000 7fae5b7fe640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae5c1544b0 0x7fae5c145450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.354+0000 7fae5b7fe640 1 -- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae5c06c0b0 msgr2=0x7fae5c0b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.354+0000 7fae5b7fe640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.354+0000 7fae5b7fe640 1 -- 172.21.15.129:0/1606505933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae54009a70 con 0x7fae5c15a2e0 2024-03-23T16:00:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.356+0000 7fae5b7fe640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 0x7fae5c0cea80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fae5402c560 tx=0x7fae54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.360+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5403e610 con 0x7fae5c15a2e0 2024-03-23T16:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.360+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5403e770 con 0x7fae5c15a2e0 2024-03-23T16:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.360+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5c1f4300 con 0x7fae5c15a2e0 2024-03-23T16:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.360+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5403eb20 con 0x7fae5c15a2e0 2024-03-23T16:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.360+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5c0d0880 con 0x7fae5c15a2e0 2024-03-23T16:00:31.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.361+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae5c1f44c0 con 0x7fae5c15a2e0 2024-03-23T16:00:31.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.381+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fae5403b050 con 0x7fae5c15a2e0 2024-03-23T16:00:31.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.382+0000 7fae437fe640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fae4806b860 0x7fae4806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:31.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.382+0000 7fae5affd640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fae4806b860 0x7fae4806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:31.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.383+0000 7fae5affd640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fae4806b860 0x7fae4806dd20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fae4c007fd0 tx=0x7fae4c007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:31.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.384+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fae5407b650 con 0x7fae5c15a2e0 2024-03-23T16:00:31.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.631+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fae540778f0 con 0x7fae5c15a2e0 2024-03-23T16:00:31.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.862+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae5c075740 con 0x7fae5c15a2e0 2024-03-23T16:00:31.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.898+0000 7fae437fe640 1 -- 172.21.15.129:0/1606505933 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fae5404e020 con 0x7fae5c15a2e0 2024-03-23T16:00:31.900 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:31.900 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:31.900 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:31.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fae4806b860 msgr2=0x7fae4806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fae4806b860 0x7fae4806dd20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fae4c007fd0 tx=0x7fae4c007e80 comp rx=0 tx=0).stop 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 msgr2=0x7fae5c0cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 0x7fae5c0cea80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fae5402c560 tx=0x7fae54004a40 comp rx=0 tx=0).stop 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 shutdown_connections 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae5c15a2e0 0x7fae5c0cea80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae5c1544b0 0x7fae5c145450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fae4806b860 0x7fae4806dd20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 --2- 172.21.15.129:0/1606505933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae5c06c0b0 0x7fae5c0b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 >> 172.21.15.129:0/1606505933 conn(0x7fae5c061b50 msgr2=0x7fae5c14a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 shutdown_connections 2024-03-23T16:00:31.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:31.900+0000 7fae6140e640 1 -- 172.21.15.129:0/1606505933 wait complete. 2024-03-23T16:00:32.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:32.519 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:33.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.620+0000 7feeeb42e640 1 Processor -- start 2024-03-23T16:00:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.621+0000 7feeeb42e640 1 -- start start 2024-03-23T16:00:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.621+0000 7feeeb42e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feee40ceb00 0x7feee415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.621+0000 7feeeb42e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.621+0000 7feeeb42e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feee415b290 con 0x7feee40ccd60 2024-03-23T16:00:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.621+0000 7feeeb42e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7feee40ce800 con 0x7feee40ceb00 2024-03-23T16:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.621+0000 7feeeb42e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feee4154a70 con 0x7feee40cef40 2024-03-23T16:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.622+0000 7feee89c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feee40ceb00 0x7feee415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.622+0000 7feee91ca640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7feee40ccd60 0x7feee406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:52260/0 (socket says 172.21.15.129:52260) 2024-03-23T16:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.622+0000 7feee91ca640 1 -- 172.21.15.129:0/116124633 learned_addr learned my addr 172.21.15.129:0/116124633 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.622+0000 7feee99cb640 1 --2- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.625+0000 7feedbfff640 1 -- 172.21.15.129:0/116124633 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221078280 0 0) 0x7feee0001110 con 0x7feee40ccd60 2024-03-23T16:00:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.625+0000 7feee89c9640 1 -- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 msgr2=0x7feee415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.625+0000 7feee89c9640 1 --2- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.625+0000 7feee89c9640 1 -- 172.21.15.129:0/116124633 >> v1:172.21.15.129:6789/0 conn(0x7feee40ccd60 legacy=0x7feee406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.625+0000 7feee89c9640 1 -- 172.21.15.129:0/116124633 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee40cd950 con 0x7feee40ceb00 2024-03-23T16:00:33.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.627+0000 7feee89c9640 1 --2- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee415ad50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feed40097a0 tx=0x7feed402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc4728ef568838a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.631+0000 7feedbfff640 1 -- 172.21.15.129:0/116124633 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4030610 con 0x7feee40ceb00 2024-03-23T16:00:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.631+0000 7feedbfff640 1 -- 172.21.15.129:0/116124633 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed4030770 con 0x7feee40ceb00 2024-03-23T16:00:33.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.631+0000 7feedbfff640 1 -- 172.21.15.129:0/116124633 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4030b00 con 0x7feee40ceb00 2024-03-23T16:00:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 -- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 msgr2=0x7feee415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 --2- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee415ad50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feed40097a0 tx=0x7feed402b960 comp rx=0 tx=0).stop 2024-03-23T16:00:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 -- 172.21.15.129:0/116124633 shutdown_connections 2024-03-23T16:00:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 --2- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 --2- 172.21.15.129:0/116124633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee415ad50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 -- 172.21.15.129:0/116124633 >> 172.21.15.129:0/116124633 conn(0x7feee40b1840 msgr2=0x7feee40b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 -- 172.21.15.129:0/116124633 shutdown_connections 2024-03-23T16:00:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.632+0000 7feeeb42e640 1 -- 172.21.15.129:0/116124633 wait complete. 2024-03-23T16:00:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.633+0000 7feeeb42e640 1 Processor -- start 2024-03-23T16:00:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.633+0000 7feeeb42e640 1 -- start start 2024-03-23T16:00:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.634+0000 7feeeb42e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feee40ccd60 0x7feee40c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.634+0000 7feeeb42e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee41f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.634+0000 7feeeb42e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee40cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.634+0000 7feeeb42e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feee40c99c0 con 0x7feee40ccd60 2024-03-23T16:00:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.634+0000 7feeeb42e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feee40c7d00 con 0x7feee40ceb00 2024-03-23T16:00:33.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.634+0000 7feeeb42e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feee40c7460 con 0x7feee40cef40 2024-03-23T16:00:33.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.635+0000 7feee91ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feee40ccd60 0x7feee40c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:33.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.635+0000 7feee99cb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee40cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:33.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.635+0000 7feee89c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee41f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:33.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.635+0000 7feee91ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feee40ccd60 0x7feee40c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43638/0 (socket says 172.21.15.129:43638) 2024-03-23T16:00:33.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.635+0000 7feee91ca640 1 -- 172.21.15.129:0/4101427062 learned_addr learned my addr 172.21.15.129:0/4101427062 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.639+0000 7feee99cb640 1 -- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 msgr2=0x7feee41f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:33.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.639+0000 7feee99cb640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee41f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:33.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.639+0000 7feee99cb640 1 -- 172.21.15.129:0/4101427062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feee40ccd60 msgr2=0x7feee40c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:33.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.639+0000 7feee99cb640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feee40ccd60 0x7feee40c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:33.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.639+0000 7feee99cb640 1 -- 172.21.15.129:0/4101427062 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed4009400 con 0x7feee40cef40 2024-03-23T16:00:33.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.641+0000 7feee99cb640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee40cb610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feedc00ebd0 tx=0x7feedc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.646+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc00cd20 con 0x7feee40cef40 2024-03-23T16:00:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.646+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feedc00f040 con 0x7feee40cef40 2024-03-23T16:00:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.646+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee40c7740 con 0x7feee40cef40 2024-03-23T16:00:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.646+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc0146c0 con 0x7feee40cef40 2024-03-23T16:00:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.646+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feee41f0f90 con 0x7feee40cef40 2024-03-23T16:00:33.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.647+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee404f530 con 0x7feee40cef40 2024-03-23T16:00:33.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.671+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feedc014820 con 0x7feee40cef40 2024-03-23T16:00:33.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.671+0000 7feed9ffb640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec806b810 0x7feec806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:33.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.671+0000 7feee91ca640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec806b810 0x7feec806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:33.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.672+0000 7feee91ca640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec806b810 0x7feec806dcd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7feee0007980 tx=0x7feee00078d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:33.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.929+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7feedc00ce80 con 0x7feee40cef40 2024-03-23T16:00:33.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:33.946+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feedc055ea0 con 0x7feee40cef40 2024-03-23T16:00:34.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.164+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feee406db50 con 0x7feee40cef40 2024-03-23T16:00:34.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.179+0000 7feed9ffb640 1 -- 172.21.15.129:0/4101427062 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7feedc018090 con 0x7feee40cef40 2024-03-23T16:00:34.182 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:34.182 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:34.182 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec806b810 msgr2=0x7feec806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec806b810 0x7feec806dcd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7feee0007980 tx=0x7feee00078d0 comp rx=0 tx=0).stop 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 msgr2=0x7feee40cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee40cb610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feedc00ebd0 tx=0x7feedc00c490 comp rx=0 tx=0).stop 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 shutdown_connections 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feee40cef40 0x7feee40cb610 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7feec806b810 0x7feec806dcd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feee40ceb00 0x7feee41f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:34.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 --2- 172.21.15.129:0/4101427062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feee40ccd60 0x7feee40c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:34.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 >> 172.21.15.129:0/4101427062 conn(0x7feee40b1840 msgr2=0x7feee4154130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:34.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 shutdown_connections 2024-03-23T16:00:34.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:34.182+0000 7feeeb42e640 1 -- 172.21.15.129:0/4101427062 wait complete. 2024-03-23T16:00:34.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:34.800 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:35.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:35.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.901+0000 7f97c516c640 1 Processor -- start 2024-03-23T16:00:35.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.901+0000 7f97c516c640 1 -- start start 2024-03-23T16:00:35.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.902+0000 7f97c516c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c01544d0 0x7f97c006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:35.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.902+0000 7f97c516c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f97c015e270 0x7f97c00ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:35.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.902+0000 7f97c516c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c015aa90 con 0x7f97c00c7ca0 2024-03-23T16:00:35.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.902+0000 7f97c516c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f97c015de50 con 0x7f97c015e270 2024-03-23T16:00:35.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.902+0000 7f97c516c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97c00c4a70 con 0x7f97c01544d0 2024-03-23T16:00:35.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.903+0000 7f97be59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f97c00c7ca0 0x7f97c015a2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:52270/0 (socket says 172.21.15.129:52270) 2024-03-23T16:00:35.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.903+0000 7f97bed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c01544d0 0x7f97c006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:35.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.903+0000 7f97be59c640 1 -- 172.21.15.129:0/1947042285 learned_addr learned my addr 172.21.15.129:0/1947042285 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:35.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.903+0000 7f97bf59e640 1 --2- 172.21.15.129:0/1947042285 >> v2:172.21.15.157:3300/0 conn(0x7f97c015e270 0x7f97c00ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:35.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.906+0000 7f97bed9d640 1 -- 172.21.15.129:0/1947042285 >> v2:172.21.15.157:3300/0 conn(0x7f97c015e270 msgr2=0x7f97c00ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:35.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.906+0000 7f97bed9d640 1 --2- 172.21.15.129:0/1947042285 >> v2:172.21.15.157:3300/0 conn(0x7f97c015e270 0x7f97c00ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:35.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.906+0000 7f97bed9d640 1 -- 172.21.15.129:0/1947042285 >> v1:172.21.15.129:6789/0 conn(0x7f97c00c7ca0 legacy=0x7f97c015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:35.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.906+0000 7f97bed9d640 1 -- 172.21.15.129:0/1947042285 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c015ad70 con 0x7f97c01544d0 2024-03-23T16:00:35.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.909+0000 7f97bed9d640 1 --2- 172.21.15.129:0/1947042285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c01544d0 0x7f97c006bb70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f97b00097a0 tx=0x7f97b002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9eb32ab18f296a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:35.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.914+0000 7f97bdd9b640 1 -- 172.21.15.129:0/1947042285 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0031830 con 0x7f97c01544d0 2024-03-23T16:00:35.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.914+0000 7f97bdd9b640 1 -- 172.21.15.129:0/1947042285 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b0031990 con 0x7f97c01544d0 2024-03-23T16:00:35.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.914+0000 7f97bdd9b640 1 -- 172.21.15.129:0/1947042285 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0031d20 con 0x7f97c01544d0 2024-03-23T16:00:35.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 -- 172.21.15.129:0/1947042285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c01544d0 msgr2=0x7f97c006bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:35.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 --2- 172.21.15.129:0/1947042285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c01544d0 0x7f97c006bb70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f97b00097a0 tx=0x7f97b002b8d0 comp rx=0 tx=0).stop 2024-03-23T16:00:35.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 -- 172.21.15.129:0/1947042285 shutdown_connections 2024-03-23T16:00:35.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 --2- 172.21.15.129:0/1947042285 >> v2:172.21.15.157:3300/0 conn(0x7f97c015e270 0x7f97c00ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:35.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 --2- 172.21.15.129:0/1947042285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c01544d0 0x7f97c006bb70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:35.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 -- 172.21.15.129:0/1947042285 >> 172.21.15.129:0/1947042285 conn(0x7f97c006e230 msgr2=0x7f97c006e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:35.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 -- 172.21.15.129:0/1947042285 shutdown_connections 2024-03-23T16:00:35.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.915+0000 7f97c516c640 1 -- 172.21.15.129:0/1947042285 wait complete. 2024-03-23T16:00:35.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 Processor -- start 2024-03-23T16:00:35.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 -- start start 2024-03-23T16:00:35.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c00c7ca0 0x7f97c01f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:35.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c01544d0 0x7f97c0152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97c015e270 0x7f97c014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97c01f0a10 con 0x7f97c01544d0 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f97c014e550 con 0x7f97c015e270 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.917+0000 7f97c516c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97c01e65b0 con 0x7f97c00c7ca0 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.918+0000 7f97be59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c01544d0 0x7f97c0152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.918+0000 7f97bed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c00c7ca0 0x7f97c01f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.918+0000 7f97bf59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97c015e270 0x7f97c014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:35.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.918+0000 7f97bed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c00c7ca0 0x7f97c01f04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:42266/0 (socket says 172.21.15.129:42266) 2024-03-23T16:00:35.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.918+0000 7f97bed9d640 1 -- 172.21.15.129:0/4244404953 learned_addr learned my addr 172.21.15.129:0/4244404953 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:35.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.922+0000 7f97be59c640 1 -- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c00c7ca0 msgr2=0x7f97c01f04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:35.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.922+0000 7f97be59c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c00c7ca0 0x7f97c01f04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:35.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.922+0000 7f97be59c640 1 -- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97c015e270 msgr2=0x7f97c014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:35.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.922+0000 7f97be59c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97c015e270 0x7f97c014e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:35.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.922+0000 7f97be59c640 1 -- 172.21.15.129:0/4244404953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b0009400 con 0x7f97c01544d0 2024-03-23T16:00:35.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.924+0000 7f97be59c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c01544d0 0x7f97c0152a20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f97b4000f60 tx=0x7f97b400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.928+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4005ce0 con 0x7f97c01544d0 2024-03-23T16:00:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.928+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b4005e40 con 0x7f97c01544d0 2024-03-23T16:00:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.928+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4007310 con 0x7f97c01544d0 2024-03-23T16:00:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.928+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c01e6850 con 0x7f97c01544d0 2024-03-23T16:00:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.929+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c01e6e10 con 0x7f97c01544d0 2024-03-23T16:00:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:35.929+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c0075740 con 0x7f97c01544d0 2024-03-23T16:00:36.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.217+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97b4007470 con 0x7f97c01544d0 2024-03-23T16:00:36.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.218+0000 7f97a77fe640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97ac06b890 0x7f97ac06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:36.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.218+0000 7f97bed9d640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97ac06b890 0x7f97ac06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:36.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.218+0000 7f97bed9d640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97ac06b890 0x7f97ac06dd50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f97b002b910 tx=0x7f97b003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:36.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.239+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f97b4007720 con 0x7f97c01544d0 2024-03-23T16:00:36.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.239+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97b4054590 con 0x7f97c01544d0 2024-03-23T16:00:36.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.457+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97c006bb50 con 0x7f97c01544d0 2024-03-23T16:00:36.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.472+0000 7f97a77fe640 1 -- 172.21.15.129:0/4244404953 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f97b4017020 con 0x7f97c01544d0 2024-03-23T16:00:36.475 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:36.475 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:36.475 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97ac06b890 msgr2=0x7f97ac06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97ac06b890 0x7f97ac06dd50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f97b002b910 tx=0x7f97b003c040 comp rx=0 tx=0).stop 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c01544d0 msgr2=0x7f97c0152a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c01544d0 0x7f97c0152a20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f97b4000f60 tx=0x7f97b400c800 comp rx=0 tx=0).stop 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 shutdown_connections 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f97ac06b890 0x7f97ac06dd50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97c015e270 0x7f97c014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c01544d0 0x7f97c0152a20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:36.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 --2- 172.21.15.129:0/4244404953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97c00c7ca0 0x7f97c01f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:36.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 >> 172.21.15.129:0/4244404953 conn(0x7f97c006e230 msgr2=0x7f97c00c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:36.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 shutdown_connections 2024-03-23T16:00:36.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:36.475+0000 7f97c516c640 1 -- 172.21.15.129:0/4244404953 wait complete. 2024-03-23T16:00:37.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:37.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:38.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:38.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.199+0000 7f4719393640 1 Processor -- start 2024-03-23T16:00:38.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.199+0000 7f4719393640 1 -- start start 2024-03-23T16:00:38.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.200+0000 7f4719393640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47141544b0 0x7f471415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:38.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.200+0000 7f4719393640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47140caa00 0x7f471406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:38.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.200+0000 7f4719393640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47140cbcb0 con 0x7f471406c0b0 2024-03-23T16:00:38.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.200+0000 7f4719393640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f47140c5be0 con 0x7f47140caa00 2024-03-23T16:00:38.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.200+0000 7f4719393640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f471415b280 con 0x7f47141544b0 2024-03-23T16:00:38.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.200+0000 7f47127fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47140caa00 0x7f471406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:38.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.201+0000 7f4712ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47141544b0 0x7f471415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:38.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.201+0000 7f47127fc640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f47140caa00 0x7f471406bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43650/0 (socket says 172.21.15.129:43650) 2024-03-23T16:00:38.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.201+0000 7f47127fc640 1 -- 172.21.15.129:0/1037397169 learned_addr learned my addr 172.21.15.129:0/1037397169 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:38.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.203+0000 7f47127fc640 1 -- 172.21.15.129:0/1037397169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47141544b0 msgr2=0x7f471415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:38.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.203+0000 7f47127fc640 1 --2- 172.21.15.129:0/1037397169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47141544b0 0x7f471415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.204+0000 7f47127fc640 1 -- 172.21.15.129:0/1037397169 >> v1:172.21.15.129:6789/0 conn(0x7f471406c0b0 legacy=0x7f47140c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:38.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.204+0000 7f47127fc640 1 -- 172.21.15.129:0/1037397169 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471415a9e0 con 0x7f47140caa00 2024-03-23T16:00:38.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.208+0000 7f47127fc640 1 --2- 172.21.15.129:0/1037397169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47140caa00 0x7f471406bb70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4708009870 tx=0x7f470802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e049614a6202b596 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:38.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.210+0000 7f4711ffb640 1 -- 172.21.15.129:0/1037397169 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470802f840 con 0x7f47140caa00 2024-03-23T16:00:38.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.212+0000 7f4711ffb640 1 -- 172.21.15.129:0/1037397169 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470802f9a0 con 0x7f47140caa00 2024-03-23T16:00:38.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.212+0000 7f4711ffb640 1 -- 172.21.15.129:0/1037397169 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470802fd30 con 0x7f47140caa00 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 -- 172.21.15.129:0/1037397169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47140caa00 msgr2=0x7f471406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 --2- 172.21.15.129:0/1037397169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47140caa00 0x7f471406bb70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4708009870 tx=0x7f470802ba20 comp rx=0 tx=0).stop 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 -- 172.21.15.129:0/1037397169 shutdown_connections 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 --2- 172.21.15.129:0/1037397169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47140caa00 0x7f471406bb70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 --2- 172.21.15.129:0/1037397169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47141544b0 0x7f471415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 -- 172.21.15.129:0/1037397169 >> 172.21.15.129:0/1037397169 conn(0x7f4714061b50 msgr2=0x7f4714061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 -- 172.21.15.129:0/1037397169 shutdown_connections 2024-03-23T16:00:38.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.213+0000 7f4719393640 1 -- 172.21.15.129:0/1037397169 wait complete. 2024-03-23T16:00:38.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.215+0000 7f4719393640 1 Processor -- start 2024-03-23T16:00:38.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.215+0000 7f4719393640 1 -- start start 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.216+0000 7f4719393640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471406c0b0 0x7f4714141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.216+0000 7f4719393640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47140caa00 0x7f47141f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.216+0000 7f4719393640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 0x7f47141562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.216+0000 7f4719393640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4714148910 con 0x7f47141544b0 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.216+0000 7f4719393640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4714146a70 con 0x7f47140caa00 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.216+0000 7f4719393640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4714155e50 con 0x7f471406c0b0 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.217+0000 7f47137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 0x7f47141562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.217+0000 7f4712ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471406c0b0 0x7f4714141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.217+0000 7f47127fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47140caa00 0x7f47141f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.217+0000 7f47137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 0x7f47141562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43654/0 (socket says 172.21.15.129:43654) 2024-03-23T16:00:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.217+0000 7f47137fe640 1 -- 172.21.15.129:0/1701624183 learned_addr learned my addr 172.21.15.129:0/1701624183 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:38.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.221+0000 7f47137fe640 1 -- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471406c0b0 msgr2=0x7f4714141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.221+0000 7f47137fe640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471406c0b0 0x7f4714141170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.221+0000 7f47137fe640 1 -- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47140caa00 msgr2=0x7f47141f3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.221+0000 7f47137fe640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47140caa00 0x7f47141f3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.221+0000 7f47137fe640 1 -- 172.21.15.129:0/1701624183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47080094d0 con 0x7f47141544b0 2024-03-23T16:00:38.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.223+0000 7f47137fe640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 0x7f47141562e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c7d0 tx=0x7f470c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:38.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.227+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c003b70 con 0x7f47141544b0 2024-03-23T16:00:38.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.227+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470c003cf0 con 0x7f47141544b0 2024-03-23T16:00:38.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.227+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47141416b0 con 0x7f47141544b0 2024-03-23T16:00:38.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.227+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c00aa90 con 0x7f47141544b0 2024-03-23T16:00:38.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.227+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4714147d90 con 0x7f47141544b0 2024-03-23T16:00:38.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.229+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4714075740 con 0x7f47141544b0 2024-03-23T16:00:38.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.252+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f470c00abf0 con 0x7f47141544b0 2024-03-23T16:00:38.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.253+0000 7f46fb7fe640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f470006b890 0x7f470006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.253+0000 7f4712ffd640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f470006b890 0x7f470006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.253+0000 7f4712ffd640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f470006b890 0x7f470006dd50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4704009aa0 tx=0x7f4704009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:38.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.517+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f470c059190 con 0x7f47141544b0 2024-03-23T16:00:38.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.518+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f470c055430 con 0x7f47141544b0 2024-03-23T16:00:38.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.750+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4714155f90 con 0x7f47141544b0 2024-03-23T16:00:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.798+0000 7f46fb7fe640 1 -- 172.21.15.129:0/1701624183 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f470c003e60 con 0x7f47141544b0 2024-03-23T16:00:38.801 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:38.801 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:38.801 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.801+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f470006b890 msgr2=0x7f470006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f470006b890 0x7f470006dd50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4704009aa0 tx=0x7f4704009190 comp rx=0 tx=0).stop 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 msgr2=0x7f47141562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 0x7f47141562e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c7d0 tx=0x7f470c00cc90 comp rx=0 tx=0).stop 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 shutdown_connections 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47141544b0 0x7f47141562e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f470006b890 0x7f470006dd50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47140caa00 0x7f47141f3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 --2- 172.21.15.129:0/1701624183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471406c0b0 0x7f4714141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 >> 172.21.15.129:0/1701624183 conn(0x7f4714061b50 msgr2=0x7f47140cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 shutdown_connections 2024-03-23T16:00:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:38.802+0000 7f4719393640 1 -- 172.21.15.129:0/1701624183 wait complete. 2024-03-23T16:00:39.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:00:39.101 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:00:39.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:39.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:40.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:40.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.525+0000 7f8ff175c640 1 Processor -- start 2024-03-23T16:00:40.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.525+0000 7f8ff175c640 1 -- start start 2024-03-23T16:00:40.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.526+0000 7f8ff175c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec1544b0 0x7f8fec15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:40.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.526+0000 7f8ff175c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8fec0caa00 0x7f8fec06bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:40.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.526+0000 7f8ff175c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fec0cbcb0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.526+0000 7f8ff175c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8fec0c5be0 con 0x7f8fec0caa00 2024-03-23T16:00:40.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.526+0000 7f8ff175c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8fec15b280 con 0x7f8fec1544b0 2024-03-23T16:00:40.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.527+0000 7f8fea7fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8fec0caa00 0x7f8fec06bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:40.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.527+0000 7f8feb7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8fec06c0b0 0x7f8fec0c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57622/0 (socket says 172.21.15.129:57622) 2024-03-23T16:00:40.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.527+0000 7f8feb7fe640 1 -- 172.21.15.129:0/3538307830 learned_addr learned my addr 172.21.15.129:0/3538307830 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:40.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.527+0000 7f8feaffd640 1 --2- 172.21.15.129:0/3538307830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec1544b0 0x7f8fec15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:40.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.530+0000 7f8fe9ffb640 1 -- 172.21.15.129:0/3538307830 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1963688776 0 0) 0x7f8fec0cbcb0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.530+0000 7f8fe9ffb640 1 -- 172.21.15.129:0/3538307830 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fd4003520 con 0x7f8fec06c0b0 2024-03-23T16:00:40.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.531+0000 7f8feaffd640 1 -- 172.21.15.129:0/3538307830 >> v2:172.21.15.157:3300/0 conn(0x7f8fec0caa00 msgr2=0x7f8fec06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:40.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.531+0000 7f8feaffd640 1 --2- 172.21.15.129:0/3538307830 >> v2:172.21.15.157:3300/0 conn(0x7f8fec0caa00 0x7f8fec06bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:40.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.531+0000 7f8feaffd640 1 -- 172.21.15.129:0/3538307830 >> v1:172.21.15.129:6789/0 conn(0x7f8fec06c0b0 legacy=0x7f8fec0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:40.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.531+0000 7f8feaffd640 1 -- 172.21.15.129:0/3538307830 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fec15a9e0 con 0x7f8fec1544b0 2024-03-23T16:00:40.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.534+0000 7f8feaffd640 1 --2- 172.21.15.129:0/3538307830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec1544b0 0x7f8fec15a2e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0097a0 tx=0x7f8fdc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70517de82afdd884 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:40.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.536+0000 7f8fe9ffb640 1 -- 172.21.15.129:0/3538307830 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc030610 con 0x7f8fec1544b0 2024-03-23T16:00:40.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.538+0000 7f8fe9ffb640 1 -- 172.21.15.129:0/3538307830 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fdc030770 con 0x7f8fec1544b0 2024-03-23T16:00:40.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.538+0000 7f8fe9ffb640 1 -- 172.21.15.129:0/3538307830 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc030ae0 con 0x7f8fec1544b0 2024-03-23T16:00:40.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 -- 172.21.15.129:0/3538307830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec1544b0 msgr2=0x7f8fec15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:40.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 --2- 172.21.15.129:0/3538307830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec1544b0 0x7f8fec15a2e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0097a0 tx=0x7f8fdc02b960 comp rx=0 tx=0).stop 2024-03-23T16:00:40.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 -- 172.21.15.129:0/3538307830 shutdown_connections 2024-03-23T16:00:40.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 --2- 172.21.15.129:0/3538307830 >> v2:172.21.15.157:3300/0 conn(0x7f8fec0caa00 0x7f8fec06bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:40.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 --2- 172.21.15.129:0/3538307830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec1544b0 0x7f8fec15a2e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:40.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 -- 172.21.15.129:0/3538307830 >> 172.21.15.129:0/3538307830 conn(0x7f8fec061b50 msgr2=0x7f8fec061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:40.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 -- 172.21.15.129:0/3538307830 shutdown_connections 2024-03-23T16:00:40.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.539+0000 7f8ff175c640 1 -- 172.21.15.129:0/3538307830 wait complete. 2024-03-23T16:00:40.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.541+0000 7f8ff175c640 1 Processor -- start 2024-03-23T16:00:40.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.541+0000 7f8ff175c640 1 -- start start 2024-03-23T16:00:40.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.542+0000 7f8ff175c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 0x7f8fec0cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:40.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.542+0000 7f8ff175c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec0caa00 0x7f8fec0c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:40.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.542+0000 7f8ff175c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fec1544b0 0x7f8fec148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:40.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.542+0000 7f8ff175c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec0cfdc0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.542+0000 7f8ff175c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec1440e0 con 0x7f8fec1544b0 2024-03-23T16:00:40.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.542+0000 7f8ff175c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8fec0c8420 con 0x7f8fec0caa00 2024-03-23T16:00:40.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.543+0000 7f8feaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 0x7f8fec0cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:40.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.543+0000 7f8fea7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec0caa00 0x7f8fec0c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:40.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.543+0000 7f8feb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fec1544b0 0x7f8fec148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:40.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.543+0000 7f8feaffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 0x7f8fec0cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58802/0 (socket says 172.21.15.129:58802) 2024-03-23T16:00:40.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.543+0000 7f8feaffd640 1 -- 172.21.15.129:0/1386292798 learned_addr learned my addr 172.21.15.129:0/1386292798 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:40.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.547+0000 7f8feaffd640 1 -- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec0caa00 msgr2=0x7f8fec0c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:40.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.547+0000 7f8feaffd640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec0caa00 0x7f8fec0c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:40.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.547+0000 7f8feaffd640 1 -- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fec1544b0 msgr2=0x7f8fec148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:40.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.547+0000 7f8feaffd640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fec1544b0 0x7f8fec148120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:40.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.547+0000 7f8feaffd640 1 -- 172.21.15.129:0/1386292798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc009400 con 0x7f8fec06c0b0 2024-03-23T16:00:40.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.549+0000 7f8feaffd640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 0x7f8fec0cf880 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000c00 tx=0x7f8fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:40.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.552+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc03e070 con 0x7f8fec06c0b0 2024-03-23T16:00:40.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.553+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fdc0044c0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.553+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc030ea0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.553+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fec0c86a0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.553+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fec0d0500 con 0x7f8fec06c0b0 2024-03-23T16:00:40.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.554+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fec147f70 con 0x7f8fec06c0b0 2024-03-23T16:00:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.572+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8fdc003850 con 0x7f8fec06c0b0 2024-03-23T16:00:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.573+0000 7f8fd37fe640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8fd806b890 0x7f8fd806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:40.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.573+0000 7f8fea7fc640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8fd806b890 0x7f8fd806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:40.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.573+0000 7f8fea7fc640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8fd806b890 0x7f8fd806dd50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8fec149a10 tx=0x7f8fe0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:40.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.577+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8fdc07abc0 con 0x7f8fec06c0b0 2024-03-23T16:00:40.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:40.826+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8fdc076e60 con 0x7f8fec06c0b0 2024-03-23T16:00:41.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.076+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fec075740 con 0x7f8fec06c0b0 2024-03-23T16:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.129+0000 7f8fd37fe640 1 -- 172.21.15.129:0/1386292798 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8fdc07a520 con 0x7f8fec06c0b0 2024-03-23T16:00:41.131 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:41.131 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:41.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8fd806b890 msgr2=0x7f8fd806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:41.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8fd806b890 0x7f8fd806dd50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8fec149a10 tx=0x7f8fe0009190 comp rx=0 tx=0).stop 2024-03-23T16:00:41.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 msgr2=0x7f8fec0cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:41.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 0x7f8fec0cf880 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000c00 tx=0x7f8fdc0046c0 comp rx=0 tx=0).stop 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 shutdown_connections 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f8fd806b890 0x7f8fd806dd50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8fec1544b0 0x7f8fec148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8fec0caa00 0x7f8fec0c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 --2- 172.21.15.129:0/1386292798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fec06c0b0 0x7f8fec0cf880 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 >> 172.21.15.129:0/1386292798 conn(0x7f8fec061b50 msgr2=0x7f8fec0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 shutdown_connections 2024-03-23T16:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:41.131+0000 7f8ff175c640 1 -- 172.21.15.129:0/1386292798 wait complete. 2024-03-23T16:00:41.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:41.749 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:42.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:42.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.851+0000 7fbce06c9640 1 Processor -- start 2024-03-23T16:00:42.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.851+0000 7fbce06c9640 1 -- start start 2024-03-23T16:00:42.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.852+0000 7fbce06c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:42.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.852+0000 7fbce06c9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbcd80ceb00 0x7fbcd806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:42.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.852+0000 7fbce06c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd815b290 con 0x7fbcd80cef40 2024-03-23T16:00:42.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.852+0000 7fbce06c9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd80ce800 con 0x7fbcd80ceb00 2024-03-23T16:00:42.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.852+0000 7fbce06c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd8154a70 con 0x7fbcd80ccd60 2024-03-23T16:00:42.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.853+0000 7fbcddc64640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbcd80ceb00 0x7fbcd806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:42.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.853+0000 7fbcde465640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:42.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.854+0000 7fbcde465640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:46008/0 (socket says 172.21.15.129:46008) 2024-03-23T16:00:42.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.854+0000 7fbcde465640 1 -- 172.21.15.129:0/3275013362 learned_addr learned my addr 172.21.15.129:0/3275013362 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:42.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.856+0000 7fbcde465640 1 -- 172.21.15.129:0/3275013362 >> v2:172.21.15.157:3300/0 conn(0x7fbcd80ceb00 msgr2=0x7fbcd806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:42.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.856+0000 7fbcde465640 1 --2- 172.21.15.129:0/3275013362 >> v2:172.21.15.157:3300/0 conn(0x7fbcd80ceb00 0x7fbcd806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:42.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.856+0000 7fbcde465640 1 -- 172.21.15.129:0/3275013362 >> v1:172.21.15.157:6789/0 conn(0x7fbcd80cef40 legacy=0x7fbcd815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:42.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.856+0000 7fbcde465640 1 -- 172.21.15.129:0/3275013362 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80cd950 con 0x7fbcd80ccd60 2024-03-23T16:00:42.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.859+0000 7fbcde465640 1 --2- 172.21.15.129:0/3275013362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd815ad50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009870 tx=0x7fbcd402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d867d0e52deb414f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:42.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.864+0000 7fbcdd463640 1 -- 172.21.15.129:0/3275013362 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd402f840 con 0x7fbcd80ccd60 2024-03-23T16:00:42.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.864+0000 7fbcdd463640 1 -- 172.21.15.129:0/3275013362 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd402f9a0 con 0x7fbcd80ccd60 2024-03-23T16:00:42.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.864+0000 7fbcdd463640 1 -- 172.21.15.129:0/3275013362 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd402fd30 con 0x7fbcd80ccd60 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 -- 172.21.15.129:0/3275013362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 msgr2=0x7fbcd815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 --2- 172.21.15.129:0/3275013362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd815ad50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009870 tx=0x7fbcd402ba20 comp rx=0 tx=0).stop 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 -- 172.21.15.129:0/3275013362 shutdown_connections 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 --2- 172.21.15.129:0/3275013362 >> v2:172.21.15.157:3300/0 conn(0x7fbcd80ceb00 0x7fbcd806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 --2- 172.21.15.129:0/3275013362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd815ad50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 -- 172.21.15.129:0/3275013362 >> 172.21.15.129:0/3275013362 conn(0x7fbcd80b1840 msgr2=0x7fbcd80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 -- 172.21.15.129:0/3275013362 shutdown_connections 2024-03-23T16:00:42.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.865+0000 7fbce06c9640 1 -- 172.21.15.129:0/3275013362 wait complete. 2024-03-23T16:00:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.867+0000 7fbce06c9640 1 Processor -- start 2024-03-23T16:00:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.867+0000 7fbce06c9640 1 -- start start 2024-03-23T16:00:42.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.868+0000 7fbce06c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd81f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.868+0000 7fbce06c9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd80ceb00 0x7fbcd815c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.868+0000 7fbce06c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd80cef40 0x7fbcd8151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.868+0000 7fbce06c9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd81f0240 con 0x7fbcd80ceb00 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.868+0000 7fbce06c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd81f07c0 con 0x7fbcd80cef40 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.868+0000 7fbce06c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd81f1190 con 0x7fbcd80ccd60 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.869+0000 7fbcde465640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd81f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.869+0000 7fbcddc64640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd80ceb00 0x7fbcd815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.869+0000 7fbcdec66640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd80cef40 0x7fbcd8151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.869+0000 7fbcde465640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd81f0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:46024/0 (socket says 172.21.15.129:46024) 2024-03-23T16:00:42.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.869+0000 7fbcde465640 1 -- 172.21.15.129:0/1116230060 learned_addr learned my addr 172.21.15.129:0/1116230060 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:42.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.873+0000 7fbcde465640 1 -- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd80cef40 msgr2=0x7fbcd8151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:42.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.873+0000 7fbcde465640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd80cef40 0x7fbcd8151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:42.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.873+0000 7fbcde465640 1 -- 172.21.15.129:0/1116230060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd80ceb00 msgr2=0x7fbcd815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:42.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.873+0000 7fbcde465640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd80ceb00 0x7fbcd815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:42.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.873+0000 7fbcde465640 1 -- 172.21.15.129:0/1116230060 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd40094d0 con 0x7fbcd80ccd60 2024-03-23T16:00:42.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.875+0000 7fbcde465640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd81f0c50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000c00 tx=0x7fbcd4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:42.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.880+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4037c40 con 0x7fbcd80ccd60 2024-03-23T16:00:42.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.880+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd4037da0 con 0x7fbcd80ccd60 2024-03-23T16:00:42.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.880+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4041640 con 0x7fbcd80ccd60 2024-03-23T16:00:42.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.880+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd81571d0 con 0x7fbcd80ccd60 2024-03-23T16:00:42.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.880+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd81f0450 con 0x7fbcd80ccd60 2024-03-23T16:00:42.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.882+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd804f530 con 0x7fbcd80ccd60 2024-03-23T16:00:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.992+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbcd40417a0 con 0x7fbcd80ccd60 2024-03-23T16:00:42.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.992+0000 7fbcceffd640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbcbc06b860 0x7fbcbc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:42.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.992+0000 7fbcddc64640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbcbc06b860 0x7fbcbc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:42.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.993+0000 7fbcddc64640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbcbc06b860 0x7fbcbc06dd20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbcd806d670 tx=0x7fbcc8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:42.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:42.998+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fbcd403c070 con 0x7fbcd80ccd60 2024-03-23T16:00:43.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.227+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbcd4078680 con 0x7fbcd80ccd60 2024-03-23T16:00:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.468+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcd80686f0 con 0x7fbcd80ccd60 2024-03-23T16:00:43.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.495+0000 7fbcceffd640 1 -- 172.21.15.129:0/1116230060 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fbcd407bd40 con 0x7fbcd80ccd60 2024-03-23T16:00:43.498 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:43.498 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:43.498 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:43.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbcbc06b860 msgr2=0x7fbcbc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:43.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbcbc06b860 0x7fbcbc06dd20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbcd806d670 tx=0x7fbcc8009190 comp rx=0 tx=0).stop 2024-03-23T16:00:43.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 msgr2=0x7fbcd81f0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:43.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd81f0c50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000c00 tx=0x7fbcd4004730 comp rx=0 tx=0).stop 2024-03-23T16:00:43.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 shutdown_connections 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbcbc06b860 0x7fbcbc06dd20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd80cef40 0x7fbcd8151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcd80ceb00 0x7fbcd815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 --2- 172.21.15.129:0/1116230060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd80ccd60 0x7fbcd81f0c50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 >> 172.21.15.129:0/1116230060 conn(0x7fbcd80b1840 msgr2=0x7fbcd8158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 shutdown_connections 2024-03-23T16:00:43.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:43.498+0000 7fbce06c9640 1 -- 172.21.15.129:0/1116230060 wait complete. 2024-03-23T16:00:44.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:44.115 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:45.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:45.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.216+0000 7fa807042640 1 Processor -- start 2024-03-23T16:00:45.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.216+0000 7fa807042640 1 -- start start 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa807042640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa8000ceb00 0x7fa80015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa807042640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa807042640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa80015b290 con 0x7fa8000ccd60 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa807042640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fa8000ce800 con 0x7fa8000ceb00 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa807042640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa800154a70 con 0x7fa8000cef40 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa7fffff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa8000ceb00 0x7fa80015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.217+0000 7fa8055df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:45.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.218+0000 7fa8055df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:47262/0 (socket says 172.21.15.129:47262) 2024-03-23T16:00:45.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.218+0000 7fa8055df640 1 -- 172.21.15.129:0/3554479623 learned_addr learned my addr 172.21.15.129:0/3554479623 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:45.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.221+0000 7fa7fffff640 1 -- 172.21.15.129:0/3554479623 >> v2:172.21.15.157:3300/0 conn(0x7fa8000cef40 msgr2=0x7fa80015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:45.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.221+0000 7fa7fffff640 1 --2- 172.21.15.129:0/3554479623 >> v2:172.21.15.157:3300/0 conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.221+0000 7fa7fffff640 1 -- 172.21.15.129:0/3554479623 >> v1:172.21.15.129:6789/0 conn(0x7fa8000ccd60 legacy=0x7fa80006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:45.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.221+0000 7fa7fffff640 1 -- 172.21.15.129:0/3554479623 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8000cd950 con 0x7fa8000ceb00 2024-03-23T16:00:45.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.225+0000 7fa7fffff640 1 --2- 172.21.15.129:0/3554479623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000ceb00 0x7fa80015ad50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa8000cf660 tx=0x7fa7ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33a39115a0bdcb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:45.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.230+0000 7fa7ff7fe640 1 -- 172.21.15.129:0/3554479623 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec030610 con 0x7fa8000ceb00 2024-03-23T16:00:45.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.230+0000 7fa7ff7fe640 1 -- 172.21.15.129:0/3554479623 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7ec030770 con 0x7fa8000ceb00 2024-03-23T16:00:45.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.230+0000 7fa7ff7fe640 1 -- 172.21.15.129:0/3554479623 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec030b00 con 0x7fa8000ceb00 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 -- 172.21.15.129:0/3554479623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000ceb00 msgr2=0x7fa80015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 --2- 172.21.15.129:0/3554479623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000ceb00 0x7fa80015ad50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa8000cf660 tx=0x7fa7ec02b960 comp rx=0 tx=0).stop 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 -- 172.21.15.129:0/3554479623 shutdown_connections 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 --2- 172.21.15.129:0/3554479623 >> v2:172.21.15.157:3300/0 conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 --2- 172.21.15.129:0/3554479623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000ceb00 0x7fa80015ad50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 -- 172.21.15.129:0/3554479623 >> 172.21.15.129:0/3554479623 conn(0x7fa8000b1840 msgr2=0x7fa8000b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 -- 172.21.15.129:0/3554479623 shutdown_connections 2024-03-23T16:00:45.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.231+0000 7fa807042640 1 -- 172.21.15.129:0/3554479623 wait complete. 2024-03-23T16:00:45.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.232+0000 7fa807042640 1 Processor -- start 2024-03-23T16:00:45.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.232+0000 7fa807042640 1 -- start start 2024-03-23T16:00:45.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.233+0000 7fa807042640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8000ccd60 0x7fa8001e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:45.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.233+0000 7fa807042640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8000ceb00 0x7fa80015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:45.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.233+0000 7fa807042640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:45.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.233+0000 7fa807042640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa8001e5350 con 0x7fa8000cef40 2024-03-23T16:00:45.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.233+0000 7fa807042640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa8001e55f0 con 0x7fa8000ceb00 2024-03-23T16:00:45.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.233+0000 7fa807042640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8000c9e80 con 0x7fa8000ccd60 2024-03-23T16:00:45.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.234+0000 7fa8055df640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:45.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.234+0000 7fa7fffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8000ceb00 0x7fa80015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:45.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.234+0000 7fa804dde640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8000ccd60 0x7fa8001e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:45.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.234+0000 7fa8055df640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58830/0 (socket says 172.21.15.129:58830) 2024-03-23T16:00:45.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.234+0000 7fa8055df640 1 -- 172.21.15.129:0/1610986290 learned_addr learned my addr 172.21.15.129:0/1610986290 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:45.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.237+0000 7fa8055df640 1 -- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8000ccd60 msgr2=0x7fa8001e5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:45.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.237+0000 7fa8055df640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8000ccd60 0x7fa8001e5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.237+0000 7fa8055df640 1 -- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8000ceb00 msgr2=0x7fa80015ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:45.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.237+0000 7fa8055df640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8000ceb00 0x7fa80015ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.237+0000 7fa8055df640 1 -- 172.21.15.129:0/1610986290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec009400 con 0x7fa8000cef40 2024-03-23T16:00:45.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.240+0000 7fa8055df640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 0x7fa80015b8f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800efc0 tx=0x7fa7f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:45.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.242+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800cd20 con 0x7fa8000cef40 2024-03-23T16:00:45.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.242+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8000c9710 con 0x7fa8000cef40 2024-03-23T16:00:45.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.242+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8000c99f0 con 0x7fa8000cef40 2024-03-23T16:00:45.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.244+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa80004f530 con 0x7fa8000cef40 2024-03-23T16:00:45.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.267+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7f800f040 con 0x7fa8000cef40 2024-03-23T16:00:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.268+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8003bb0 con 0x7fa8000cef40 2024-03-23T16:00:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.269+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa7f8003d50 con 0x7fa8000cef40 2024-03-23T16:00:45.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.269+0000 7fa7fd7fa640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa7e406b810 0x7fa7e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:45.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.269+0000 7fa804dde640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa7e406b810 0x7fa7e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:45.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.270+0000 7fa804dde640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa7e406b810 0x7fa7e406dcd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40016e0 tx=0x7fa7f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:45.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.271+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa7f8059ce0 con 0x7fa8000cef40 2024-03-23T16:00:45.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.537+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa7f8055f80 con 0x7fa8000cef40 2024-03-23T16:00:45.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.777+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa80006db50 con 0x7fa8000cef40 2024-03-23T16:00:45.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.800+0000 7fa7fd7fa640 1 -- 172.21.15.129:0/1610986290 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa7f8059640 con 0x7fa8000cef40 2024-03-23T16:00:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:45.802 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:45.802 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa7e406b810 msgr2=0x7fa7e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa7e406b810 0x7fa7e406dcd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40016e0 tx=0x7fa7f4009040 comp rx=0 tx=0).stop 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 msgr2=0x7fa80015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 0x7fa80015b8f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800efc0 tx=0x7fa7f800c490 comp rx=0 tx=0).stop 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 shutdown_connections 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8000cef40 0x7fa80015b8f0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa7e406b810 0x7fa7e406dcd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8000ceb00 0x7fa80015ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 --2- 172.21.15.129:0/1610986290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8000ccd60 0x7fa8001e5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.802+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 >> 172.21.15.129:0/1610986290 conn(0x7fa8000b1840 msgr2=0x7fa800153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.803+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 shutdown_connections 2024-03-23T16:00:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:45.803+0000 7fa807042640 1 -- 172.21.15.129:0/1610986290 wait complete. 2024-03-23T16:00:46.422 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:46.422 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:47.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:47.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.524+0000 7f7ef5207640 1 Processor -- start 2024-03-23T16:00:47.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.524+0000 7f7ef5207640 1 -- start start 2024-03-23T16:00:47.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.525+0000 7f7ef5207640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7ef00ccd60 0x7f7ef015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.525+0000 7f7ef5207640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00ceb00 0x7f7ef006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.525+0000 7f7ef5207640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7ef00cef40 0x7f7ef006c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.525+0000 7f7ef5207640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef015b290 con 0x7f7ef00ccd60 2024-03-23T16:00:47.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.525+0000 7f7ef5207640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef00ce800 con 0x7f7ef00cef40 2024-03-23T16:00:47.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.525+0000 7f7ef5207640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef0154a70 con 0x7f7ef00ceb00 2024-03-23T16:00:47.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.526+0000 7f7eeed9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7ef00ccd60 0x7f7ef015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.526+0000 7f7eef59e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7ef00cef40 0x7f7ef006c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.526+0000 7f7eee59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00ceb00 0x7f7ef006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.526+0000 7f7eeed9d640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f7ef00ccd60 0x7f7ef015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58836/0 (socket says 172.21.15.129:58836) 2024-03-23T16:00:47.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.526+0000 7f7eeed9d640 1 -- 172.21.15.129:0/2132076085 learned_addr learned my addr 172.21.15.129:0/2132076085 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:47.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.529+0000 7f7eeed9d640 1 -- 172.21.15.129:0/2132076085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00ceb00 msgr2=0x7f7ef006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:47.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.529+0000 7f7eeed9d640 1 --2- 172.21.15.129:0/2132076085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00ceb00 0x7f7ef006c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.529+0000 7f7eeed9d640 1 -- 172.21.15.129:0/2132076085 >> v2:172.21.15.157:3300/0 conn(0x7f7ef00cef40 msgr2=0x7f7ef006c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:47.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.529+0000 7f7eeed9d640 1 --2- 172.21.15.129:0/2132076085 >> v2:172.21.15.157:3300/0 conn(0x7f7ef00cef40 0x7f7ef006c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.529+0000 7f7eeed9d640 1 -- 172.21.15.129:0/2132076085 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00cd950 con 0x7f7ef00ccd60 2024-03-23T16:00:47.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.532+0000 7f7eeed9d640 1 --2- 172.21.15.129:0/2132076085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef015ad50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ef015e8e0 tx=0x7f7ee002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2765025b93682ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:47.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.536+0000 7f7eedd9b640 1 -- 172.21.15.129:0/2132076085 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0030860 con 0x7f7ef00ccd60 2024-03-23T16:00:47.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.536+0000 7f7eedd9b640 1 -- 172.21.15.129:0/2132076085 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee00309c0 con 0x7f7ef00ccd60 2024-03-23T16:00:47.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.536+0000 7f7eedd9b640 1 -- 172.21.15.129:0/2132076085 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0030d50 con 0x7f7ef00ccd60 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 -- 172.21.15.129:0/2132076085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 msgr2=0x7f7ef015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 --2- 172.21.15.129:0/2132076085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef015ad50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ef015e8e0 tx=0x7f7ee002b9a0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 -- 172.21.15.129:0/2132076085 shutdown_connections 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 --2- 172.21.15.129:0/2132076085 >> v2:172.21.15.157:3300/0 conn(0x7f7ef00cef40 0x7f7ef006c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 --2- 172.21.15.129:0/2132076085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00ceb00 0x7f7ef006c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 --2- 172.21.15.129:0/2132076085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef015ad50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 -- 172.21.15.129:0/2132076085 >> 172.21.15.129:0/2132076085 conn(0x7f7ef00b1840 msgr2=0x7f7ef00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 -- 172.21.15.129:0/2132076085 shutdown_connections 2024-03-23T16:00:47.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.537+0000 7f7ef5207640 1 -- 172.21.15.129:0/2132076085 wait complete. 2024-03-23T16:00:47.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.539+0000 7f7ef5207640 1 Processor -- start 2024-03-23T16:00:47.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.539+0000 7f7ef5207640 1 -- start start 2024-03-23T16:00:47.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7ef5207640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef0153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7ef5207640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef00ceb00 0x7f7ef0152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7ef5207640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00cef40 0x7f7ef0153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7ef5207640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00c7890 con 0x7f7ef00ccd60 2024-03-23T16:00:47.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7ef5207640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00c9ac0 con 0x7f7ef00ceb00 2024-03-23T16:00:47.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7ef5207640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef00c8ef0 con 0x7f7ef00cef40 2024-03-23T16:00:47.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7eeed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef0153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7eee59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef00ceb00 0x7f7ef0152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.540+0000 7f7eef59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00cef40 0x7f7ef0153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.541+0000 7f7eeed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef0153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58850/0 (socket says 172.21.15.129:58850) 2024-03-23T16:00:47.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.541+0000 7f7eeed9d640 1 -- 172.21.15.129:0/3901269589 learned_addr learned my addr 172.21.15.129:0/3901269589 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:47.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.544+0000 7f7eeed9d640 1 -- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00cef40 msgr2=0x7f7ef0153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:47.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.544+0000 7f7eeed9d640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00cef40 0x7f7ef0153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.544+0000 7f7eeed9d640 1 -- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef00ceb00 msgr2=0x7f7ef0152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:47.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.544+0000 7f7eeed9d640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef00ceb00 0x7f7ef0152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:47.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.544+0000 7f7eeed9d640 1 -- 172.21.15.129:0/3901269589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee00094d0 con 0x7f7ef00ccd60 2024-03-23T16:00:47.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.546+0000 7f7eeed9d640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef0153f70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:47.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.550+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0039a90 con 0x7f7ef00ccd60 2024-03-23T16:00:47.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.550+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee0039c10 con 0x7f7ef00ccd60 2024-03-23T16:00:47.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.550+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0043410 con 0x7f7ef00ccd60 2024-03-23T16:00:47.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.550+0000 7f7ef5207640 1 -- 172.21.15.129:0/3901269589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef00c9190 con 0x7f7ef00ccd60 2024-03-23T16:00:47.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.550+0000 7f7ef5207640 1 -- 172.21.15.129:0/3901269589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef00c9750 con 0x7f7ef00ccd60 2024-03-23T16:00:47.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.551+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec8005290 con 0x7f7ef00ccd60 2024-03-23T16:00:47.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.807+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7ee004c3b0 con 0x7f7ef00ccd60 2024-03-23T16:00:47.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.808+0000 7f7ed77fe640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7edc06b860 0x7f7edc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:47.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.808+0000 7f7eee59c640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7edc06b860 0x7f7edc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:47.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.808+0000 7f7eee59c640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7edc06b860 0x7f7edc06dd20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ef015e280 tx=0x7f7ee4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:47.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.821+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7ee007b600 con 0x7f7ef00ccd60 2024-03-23T16:00:47.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:47.822+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7ee00778a0 con 0x7f7ef00ccd60 2024-03-23T16:00:48.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.085+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ec80058f0 con 0x7f7ef00ccd60 2024-03-23T16:00:48.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.105+0000 7f7ed77fe640 1 -- 172.21.15.129:0/3901269589 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f7ec80058f0 con 0x7f7ef00ccd60 2024-03-23T16:00:48.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:48.108 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:48.108 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.107+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7edc06b860 msgr2=0x7f7edc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.107+0000 7f7ed57fa640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7edc06b860 0x7f7edc06dd20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ef015e280 tx=0x7f7ee4009190 comp rx=0 tx=0).stop 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 msgr2=0x7f7ef0153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef0153f70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0004730 comp rx=0 tx=0).stop 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 shutdown_connections 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef00cef40 0x7f7ef0153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7edc06b860 0x7f7edc06dd20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef00ceb00 0x7f7ef0152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 --2- 172.21.15.129:0/3901269589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef00ccd60 0x7f7ef0153f70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 >> 172.21.15.129:0/3901269589 conn(0x7f7ef00b1840 msgr2=0x7f7ef00cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 shutdown_connections 2024-03-23T16:00:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:48.108+0000 7f7ed57fa640 1 -- 172.21.15.129:0/3901269589 wait complete. 2024-03-23T16:00:48.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:48.725 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:49.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:49.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.826+0000 7f82758df640 1 Processor -- start 2024-03-23T16:00:49.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.826+0000 7f82758df640 1 -- start start 2024-03-23T16:00:49.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.827+0000 7f82758df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f827015a2e0 0x7f827006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:49.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.827+0000 7f82758df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f827006c0b0 0x7f82700c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:49.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.827+0000 7f82758df640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82700cbcb0 con 0x7f82701544b0 2024-03-23T16:00:49.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.827+0000 7f82758df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f82700c5be0 con 0x7f827015a2e0 2024-03-23T16:00:49.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.827+0000 7f82758df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f827015b280 con 0x7f827006c0b0 2024-03-23T16:00:49.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.828+0000 7f826effd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f82701544b0 0x7f82700caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:57638/0 (socket says 172.21.15.129:57638) 2024-03-23T16:00:49.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.828+0000 7f826effd640 1 -- 172.21.15.129:0/1731384100 learned_addr learned my addr 172.21.15.129:0/1731384100 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:49.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.828+0000 7f826f7fe640 1 --2- 172.21.15.129:0/1731384100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f827006c0b0 0x7f82700c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:49.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.828+0000 7f8267fff640 1 --2- 172.21.15.129:0/1731384100 >> v2:172.21.15.157:3300/0 conn(0x7f827015a2e0 0x7f827006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.831+0000 7f826e7fc640 1 -- 172.21.15.129:0/1731384100 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778426820 0 0) 0x7f82700cbcb0 con 0x7f82701544b0 2024-03-23T16:00:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.831+0000 7f826e7fc640 1 -- 172.21.15.129:0/1731384100 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f825c003520 con 0x7f82701544b0 2024-03-23T16:00:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.832+0000 7f826f7fe640 1 -- 172.21.15.129:0/1731384100 >> v2:172.21.15.157:3300/0 conn(0x7f827015a2e0 msgr2=0x7f827006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.832+0000 7f826f7fe640 1 --2- 172.21.15.129:0/1731384100 >> v2:172.21.15.157:3300/0 conn(0x7f827015a2e0 0x7f827006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.832+0000 7f826f7fe640 1 -- 172.21.15.129:0/1731384100 >> v1:172.21.15.129:6789/0 conn(0x7f82701544b0 legacy=0x7f82700caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.832+0000 7f826f7fe640 1 -- 172.21.15.129:0/1731384100 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827015a9e0 con 0x7f827006c0b0 2024-03-23T16:00:49.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.834+0000 7f826f7fe640 1 --2- 172.21.15.129:0/1731384100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f827006c0b0 0x7f82700c6000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8268009d90 tx=0x7f826802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74b88aed2c712f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:49.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.837+0000 7f826e7fc640 1 -- 172.21.15.129:0/1731384100 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826802cce0 con 0x7f827006c0b0 2024-03-23T16:00:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.839+0000 7f826e7fc640 1 -- 172.21.15.129:0/1731384100 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826802ce40 con 0x7f827006c0b0 2024-03-23T16:00:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.839+0000 7f826e7fc640 1 -- 172.21.15.129:0/1731384100 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268034640 con 0x7f827006c0b0 2024-03-23T16:00:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 -- 172.21.15.129:0/1731384100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f827006c0b0 msgr2=0x7f82700c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 --2- 172.21.15.129:0/1731384100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f827006c0b0 0x7f82700c6000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8268009d90 tx=0x7f826802bf30 comp rx=0 tx=0).stop 2024-03-23T16:00:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 -- 172.21.15.129:0/1731384100 shutdown_connections 2024-03-23T16:00:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 --2- 172.21.15.129:0/1731384100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f827006c0b0 0x7f82700c6000 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:49.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 --2- 172.21.15.129:0/1731384100 >> v2:172.21.15.157:3300/0 conn(0x7f827015a2e0 0x7f827006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 -- 172.21.15.129:0/1731384100 >> 172.21.15.129:0/1731384100 conn(0x7f8270061b50 msgr2=0x7f8270061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 -- 172.21.15.129:0/1731384100 shutdown_connections 2024-03-23T16:00:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.840+0000 7f82758df640 1 -- 172.21.15.129:0/1731384100 wait complete. 2024-03-23T16:00:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.841+0000 7f82758df640 1 Processor -- start 2024-03-23T16:00:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.841+0000 7f82758df640 1 -- start start 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f82758df640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 0x7f8270148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f82758df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82701544b0 0x7f8270155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f82758df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f827015a2e0 0x7f8270147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f82758df640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8270147510 con 0x7f827006c0b0 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f82758df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f82700cdd10 con 0x7f827015a2e0 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f82758df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f82701578d0 con 0x7f82701544b0 2024-03-23T16:00:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.842+0000 7f826effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 0x7f8270148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.843+0000 7f8267fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82701544b0 0x7f8270155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.843+0000 7f826f7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f827015a2e0 0x7f8270147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.843+0000 7f826effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 0x7f8270148c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58860/0 (socket says 172.21.15.129:58860) 2024-03-23T16:00:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.843+0000 7f826effd640 1 -- 172.21.15.129:0/837041448 learned_addr learned my addr 172.21.15.129:0/837041448 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.846+0000 7f826effd640 1 -- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82701544b0 msgr2=0x7f8270155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.846+0000 7f826effd640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82701544b0 0x7f8270155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.846+0000 7f826effd640 1 -- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f827015a2e0 msgr2=0x7f8270147c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.846+0000 7f826effd640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f827015a2e0 0x7f8270147c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:49.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.846+0000 7f826effd640 1 -- 172.21.15.129:0/837041448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8268009a70 con 0x7f827006c0b0 2024-03-23T16:00:49.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.851+0000 7f826effd640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 0x7f8270148c50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f826000c430 tx=0x7f826000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:49.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.860+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260005ce0 con 0x7f827006c0b0 2024-03-23T16:00:49.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.860+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8260005e40 con 0x7f827006c0b0 2024-03-23T16:00:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.860+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8270147230 con 0x7f827006c0b0 2024-03-23T16:00:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.860+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82600072f0 con 0x7f827006c0b0 2024-03-23T16:00:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.860+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82701558a0 con 0x7f827006c0b0 2024-03-23T16:00:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.862+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8270075740 con 0x7f827006c0b0 2024-03-23T16:00:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.886+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8260007450 con 0x7f827006c0b0 2024-03-23T16:00:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.886+0000 7f826cff9640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f825006b860 0x7f825006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.886+0000 7f8267fff640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f825006b860 0x7f825006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:49.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:49.887+0000 7f8267fff640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f825006b860 0x7f825006dd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8270154a40 tx=0x7f8258009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:50.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.158+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f8260058450 con 0x7f827006c0b0 2024-03-23T16:00:50.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.158+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82600546f0 con 0x7f827006c0b0 2024-03-23T16:00:50.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.379+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8270157d00 con 0x7f827006c0b0 2024-03-23T16:00:50.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.394+0000 7f826cff9640 1 -- 172.21.15.129:0/837041448 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f8260017070 con 0x7f827006c0b0 2024-03-23T16:00:50.397 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:50.397 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:50.397 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f825006b860 msgr2=0x7f825006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f825006b860 0x7f825006dd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8270154a40 tx=0x7f8258009190 comp rx=0 tx=0).stop 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 msgr2=0x7f8270148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 0x7f8270148c50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f826000c430 tx=0x7f826000c8f0 comp rx=0 tx=0).stop 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 shutdown_connections 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f825006b860 0x7f825006dd20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:50.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f827015a2e0 0x7f8270147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:50.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82701544b0 0x7f8270155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:50.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 --2- 172.21.15.129:0/837041448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f827006c0b0 0x7f8270148c50 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:50.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 >> 172.21.15.129:0/837041448 conn(0x7f8270061b50 msgr2=0x7f82701533a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:50.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 shutdown_connections 2024-03-23T16:00:50.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:50.397+0000 7f82758df640 1 -- 172.21.15.129:0/837041448 wait complete. 2024-03-23T16:00:51.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:51.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:52.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:52.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.115+0000 7f960b7c7640 1 Processor -- start 2024-03-23T16:00:52.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.115+0000 7f960b7c7640 1 -- start start 2024-03-23T16:00:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.116+0000 7f960b7c7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f960415d9c0 0x7f960406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.116+0000 7f960b7c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 0x7f96040cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.116+0000 7f960b7c7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040cb810 con 0x7f960415dda0 2024-03-23T16:00:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.116+0000 7f960b7c7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f96040cac00 con 0x7f960415d9c0 2024-03-23T16:00:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.116+0000 7f960b7c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96040caee0 con 0x7f96040ca4a0 2024-03-23T16:00:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.117+0000 7f9609563640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f960415d9c0 0x7f960406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.117+0000 7f9608d62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 0x7f96040cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.117+0000 7f9608d62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 0x7f96040cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34238/0 (socket says 172.21.15.129:34238) 2024-03-23T16:00:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.117+0000 7f9608d62640 1 -- 172.21.15.129:0/3007706318 learned_addr learned my addr 172.21.15.129:0/3007706318 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:52.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.121+0000 7f9608d62640 1 -- 172.21.15.129:0/3007706318 >> v2:172.21.15.157:3300/0 conn(0x7f960415d9c0 msgr2=0x7f960406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:52.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.121+0000 7f9608d62640 1 --2- 172.21.15.129:0/3007706318 >> v2:172.21.15.157:3300/0 conn(0x7f960415d9c0 0x7f960406c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.121+0000 7f9608d62640 1 -- 172.21.15.129:0/3007706318 >> v1:172.21.15.157:6789/0 conn(0x7f960415dda0 legacy=0x7f960415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:52.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.121+0000 7f9608d62640 1 -- 172.21.15.129:0/3007706318 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96040cb1c0 con 0x7f96040ca4a0 2024-03-23T16:00:52.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.124+0000 7f9608d62640 1 --2- 172.21.15.129:0/3007706318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 0x7f96040cbca0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fbc9500acc3fe17 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:52.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.127+0000 7f95fbfff640 1 -- 172.21.15.129:0/3007706318 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4030840 con 0x7f96040ca4a0 2024-03-23T16:00:52.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.129+0000 7f95fbfff640 1 -- 172.21.15.129:0/3007706318 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f40309a0 con 0x7f96040ca4a0 2024-03-23T16:00:52.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.129+0000 7f95fbfff640 1 -- 172.21.15.129:0/3007706318 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4030d30 con 0x7f96040ca4a0 2024-03-23T16:00:52.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 -- 172.21.15.129:0/3007706318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 msgr2=0x7f96040cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 --2- 172.21.15.129:0/3007706318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 0x7f96040cbca0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f402b980 comp rx=0 tx=0).stop 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 -- 172.21.15.129:0/3007706318 shutdown_connections 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 --2- 172.21.15.129:0/3007706318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96040ca4a0 0x7f96040cbca0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 --2- 172.21.15.129:0/3007706318 >> v2:172.21.15.157:3300/0 conn(0x7f960415d9c0 0x7f960406c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 -- 172.21.15.129:0/3007706318 >> 172.21.15.129:0/3007706318 conn(0x7f96040cfb20 msgr2=0x7f96040cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 -- 172.21.15.129:0/3007706318 shutdown_connections 2024-03-23T16:00:52.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.130+0000 7f960b7c7640 1 -- 172.21.15.129:0/3007706318 wait complete. 2024-03-23T16:00:52.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.131+0000 7f960b7c7640 1 Processor -- start 2024-03-23T16:00:52.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.131+0000 7f960b7c7640 1 -- start start 2024-03-23T16:00:52.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f960b7c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96040ca4a0 0x7f96041efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:52.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f960b7c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f960415d9c0 0x7f96040c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:52.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f960b7c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f960415dda0 0x7f96041f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:52.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f960b7c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9604153f50 con 0x7f96040ca4a0 2024-03-23T16:00:52.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f960b7c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96040c4c50 con 0x7f960415d9c0 2024-03-23T16:00:52.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f960b7c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96040c4ec0 con 0x7f960415dda0 2024-03-23T16:00:52.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f9608d62640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f960415d9c0 0x7f96040c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:52.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f9609d64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f960415dda0 0x7f96041f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:52.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.132+0000 7f9609563640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96040ca4a0 0x7f96041efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:52.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.133+0000 7f9609d64640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f960415dda0 0x7f96041f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34248/0 (socket says 172.21.15.129:34248) 2024-03-23T16:00:52.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.133+0000 7f9609d64640 1 -- 172.21.15.129:0/458726343 learned_addr learned my addr 172.21.15.129:0/458726343 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:52.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.136+0000 7f9609563640 1 -- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f960415dda0 msgr2=0x7f96041f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:52.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.136+0000 7f9609563640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f960415dda0 0x7f96041f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.136+0000 7f9609563640 1 -- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f960415d9c0 msgr2=0x7f96040c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:52.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.136+0000 7f9609563640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f960415d9c0 0x7f96040c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.136+0000 7f9609563640 1 -- 172.21.15.129:0/458726343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f40094d0 con 0x7f96040ca4a0 2024-03-23T16:00:52.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.138+0000 7f9609563640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96040ca4a0 0x7f96041efb20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f960000e8a0 tx=0x7f960000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:52.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.143+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600009e60 con 0x7f96040ca4a0 2024-03-23T16:00:52.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.143+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f960000c940 con 0x7f96040ca4a0 2024-03-23T16:00:52.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.143+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960000ccd0 con 0x7f96040ca4a0 2024-03-23T16:00:52.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.143+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96040c51a0 con 0x7f96040ca4a0 2024-03-23T16:00:52.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.143+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96040c5760 con 0x7f96040ca4a0 2024-03-23T16:00:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.144+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9604075740 con 0x7f96040ca4a0 2024-03-23T16:00:52.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.399+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9600013470 con 0x7f96040ca4a0 2024-03-23T16:00:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.399+0000 7f95f9ffb640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95e806b810 0x7f95e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.399+0000 7f9608d62640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95e806b810 0x7f95e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:52.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.400+0000 7f9608d62640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95e806b810 0x7f95e806dcd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f95f4036000 tx=0x7f95f4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:52.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.412+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9600014e40 con 0x7f96040ca4a0 2024-03-23T16:00:52.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.412+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9600055420 con 0x7f96040ca4a0 2024-03-23T16:00:52.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.632+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96040d09c0 con 0x7f96040ca4a0 2024-03-23T16:00:52.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.715+0000 7f95f9ffb640 1 -- 172.21.15.129:0/458726343 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f9600021090 con 0x7f96040ca4a0 2024-03-23T16:00:52.718 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:52.718 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:52.718 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95e806b810 msgr2=0x7f95e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95e806b810 0x7f95e806dcd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f95f4036000 tx=0x7f95f4008030 comp rx=0 tx=0).stop 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96040ca4a0 msgr2=0x7f96041efb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96040ca4a0 0x7f96041efb20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f960000e8a0 tx=0x7f960000ed60 comp rx=0 tx=0).stop 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 shutdown_connections 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f960415dda0 0x7f96041f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f95e806b810 0x7f95e806dcd0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f960415d9c0 0x7f96040c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 --2- 172.21.15.129:0/458726343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96040ca4a0 0x7f96041efb20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:52.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 >> 172.21.15.129:0/458726343 conn(0x7f96040cfb20 msgr2=0x7f96040ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:52.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 shutdown_connections 2024-03-23T16:00:52.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:52.717+0000 7f960b7c7640 1 -- 172.21.15.129:0/458726343 wait complete. 2024-03-23T16:00:53.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:53.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:54.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.437+0000 7fa4a1255640 1 Processor -- start 2024-03-23T16:00:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.437+0000 7fa4a1255640 1 -- start start 2024-03-23T16:00:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.438+0000 7fa4a1255640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.438+0000 7fa4a1255640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa49c0ceb00 0x7fa49c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.438+0000 7fa4a1255640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa49c15b290 con 0x7fa49c0cef40 2024-03-23T16:00:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.439+0000 7fa4a1255640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fa49c0ce800 con 0x7fa49c0ceb00 2024-03-23T16:00:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.439+0000 7fa4a1255640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa49c154a70 con 0x7fa49c0ccd60 2024-03-23T16:00:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.439+0000 7fa49ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.439+0000 7fa49a59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fa49c0ceb00 0x7fa49c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.439+0000 7fa49ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34266/0 (socket says 172.21.15.129:34266) 2024-03-23T16:00:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.439+0000 7fa49ad9d640 1 -- 172.21.15.129:0/1666239887 learned_addr learned my addr 172.21.15.129:0/1666239887 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:54.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.442+0000 7fa49ad9d640 1 -- 172.21.15.129:0/1666239887 >> v2:172.21.15.129:3300/0 conn(0x7fa49c0ceb00 msgr2=0x7fa49c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:54.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.442+0000 7fa49ad9d640 1 --2- 172.21.15.129:0/1666239887 >> v2:172.21.15.129:3300/0 conn(0x7fa49c0ceb00 0x7fa49c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:54.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.442+0000 7fa49ad9d640 1 -- 172.21.15.129:0/1666239887 >> v1:172.21.15.129:6789/0 conn(0x7fa49c0cef40 legacy=0x7fa49c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:54.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.442+0000 7fa49ad9d640 1 -- 172.21.15.129:0/1666239887 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49c0cd950 con 0x7fa49c0ccd60 2024-03-23T16:00:54.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.444+0000 7fa49ad9d640 1 --2- 172.21.15.129:0/1666239887 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c15ad50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009870 tx=0x7fa48c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a275acc92d93dde server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:54.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa499d9b640 1 -- 172.21.15.129:0/1666239887 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c02f840 con 0x7fa49c0ccd60 2024-03-23T16:00:54.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa499d9b640 1 -- 172.21.15.129:0/1666239887 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c02f9a0 con 0x7fa49c0ccd60 2024-03-23T16:00:54.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa499d9b640 1 -- 172.21.15.129:0/1666239887 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c02fd30 con 0x7fa49c0ccd60 2024-03-23T16:00:54.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 -- 172.21.15.129:0/1666239887 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 msgr2=0x7fa49c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 --2- 172.21.15.129:0/1666239887 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c15ad50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009870 tx=0x7fa48c02ba20 comp rx=0 tx=0).stop 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 -- 172.21.15.129:0/1666239887 shutdown_connections 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 --2- 172.21.15.129:0/1666239887 >> v2:172.21.15.129:3300/0 conn(0x7fa49c0ceb00 0x7fa49c06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 --2- 172.21.15.129:0/1666239887 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c15ad50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 -- 172.21.15.129:0/1666239887 >> 172.21.15.129:0/1666239887 conn(0x7fa49c0b1840 msgr2=0x7fa49c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 -- 172.21.15.129:0/1666239887 shutdown_connections 2024-03-23T16:00:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.448+0000 7fa4a1255640 1 -- 172.21.15.129:0/1666239887 wait complete. 2024-03-23T16:00:54.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.450+0000 7fa4a1255640 1 Processor -- start 2024-03-23T16:00:54.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.450+0000 7fa4a1255640 1 -- start start 2024-03-23T16:00:54.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa4a1255640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c1f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:54.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa4a1255640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa49c0ceb00 0x7fa49c15c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:54.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa4a1255640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa49c0cef40 0x7fa49c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:54.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa4a1255640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1f0240 con 0x7fa49c0cef40 2024-03-23T16:00:54.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa4a1255640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1f07c0 con 0x7fa49c0ceb00 2024-03-23T16:00:54.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa4a1255640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa49c1f1190 con 0x7fa49c0ccd60 2024-03-23T16:00:54.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa49ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c1f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:54.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa49b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa49c0cef40 0x7fa49c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:54.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa49ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c1f0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34268/0 (socket says 172.21.15.129:34268) 2024-03-23T16:00:54.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa49ad9d640 1 -- 172.21.15.129:0/2345860139 learned_addr learned my addr 172.21.15.129:0/2345860139 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:54.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.451+0000 7fa49a59c640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa49c0ceb00 0x7fa49c15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:54.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.455+0000 7fa49b59e640 1 -- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 msgr2=0x7fa49c1f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:54.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.455+0000 7fa49b59e640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c1f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:54.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.455+0000 7fa49b59e640 1 -- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa49c0ceb00 msgr2=0x7fa49c15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:54.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.455+0000 7fa49b59e640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa49c0ceb00 0x7fa49c15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:54.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.455+0000 7fa49b59e640 1 -- 172.21.15.129:0/2345860139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48c0094d0 con 0x7fa49c0cef40 2024-03-23T16:00:54.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.458+0000 7fa49b59e640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa49c0cef40 0x7fa49c151d70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c700 tx=0x7fa49400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:54.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.463+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494003b70 con 0x7fa49c0cef40 2024-03-23T16:00:54.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.463+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa494003cd0 con 0x7fa49c0cef40 2024-03-23T16:00:54.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.463+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494005310 con 0x7fa49c0cef40 2024-03-23T16:00:54.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.463+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49c1571d0 con 0x7fa49c0cef40 2024-03-23T16:00:54.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.463+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49c1f0450 con 0x7fa49c0cef40 2024-03-23T16:00:54.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.464+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa49c1f0610 con 0x7fa49c0cef40 2024-03-23T16:00:54.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.482+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa494005470 con 0x7fa49c0cef40 2024-03-23T16:00:54.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.482+0000 7fa4837fe640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa48806b860 0x7fa48806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:54.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.483+0000 7fa49ad9d640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa48806b860 0x7fa48806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:54.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.483+0000 7fa49ad9d640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa48806b860 0x7fa48806dd20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa48c02ba60 tx=0x7fa48c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:54.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.750+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fa494058e50 con 0x7fa49c0cef40 2024-03-23T16:00:54.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.750+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa4940550f0 con 0x7fa49c0cef40 2024-03-23T16:00:54.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.968+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa49c0686f0 con 0x7fa49c0cef40 2024-03-23T16:00:54.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.996+0000 7fa4837fe640 1 -- 172.21.15.129:0/2345860139 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fa4940587b0 con 0x7fa49c0cef40 2024-03-23T16:00:54.999 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:54.999 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:54.999 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:55.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa48806b860 msgr2=0x7fa48806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:55.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa48806b860 0x7fa48806dd20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa48c02ba60 tx=0x7fa48c008030 comp rx=0 tx=0).stop 2024-03-23T16:00:55.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa49c0cef40 msgr2=0x7fa49c151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:55.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa49c0cef40 0x7fa49c151d70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c700 tx=0x7fa49400cbc0 comp rx=0 tx=0).stop 2024-03-23T16:00:55.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 shutdown_connections 2024-03-23T16:00:55.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa49c0cef40 0x7fa49c151d70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:55.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fa48806b860 0x7fa48806dd20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:55.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa49c0ceb00 0x7fa49c15c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:55.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 --2- 172.21.15.129:0/2345860139 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa49c0ccd60 0x7fa49c1f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:55.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 >> 172.21.15.129:0/2345860139 conn(0x7fa49c0b1840 msgr2=0x7fa49c158140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:55.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 shutdown_connections 2024-03-23T16:00:55.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:54.999+0000 7fa4a1255640 1 -- 172.21.15.129:0/2345860139 wait complete. 2024-03-23T16:00:55.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:55.616 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:56.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.717+0000 7f55f0e84640 1 Processor -- start 2024-03-23T16:00:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.717+0000 7f55f0e84640 1 -- start start 2024-03-23T16:00:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55f0e84640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f55ec1544b0 0x7f55ec15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55f0e84640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ec0cbcb0 con 0x7f55ec0c6000 2024-03-23T16:00:56.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55f0e84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ec15b280 con 0x7f55ec0caa00 2024-03-23T16:00:56.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55f0e84640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f55ec0c5be0 con 0x7f55ec1544b0 2024-03-23T16:00:56.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55ea59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f55ec1544b0 0x7f55ec15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:56.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55e3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f55ec0caa00 0x7f55ec06bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:60966/0 (socket says 172.21.15.129:60966) 2024-03-23T16:00:56.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.719+0000 7f55e3fff640 1 -- 172.21.15.129:0/3296588287 learned_addr learned my addr 172.21.15.129:0/3296588287 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:00:56.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.722+0000 7f55ea59c640 1 -- 172.21.15.129:0/3296588287 >> v1:172.21.15.157:6789/0 conn(0x7f55ec0caa00 legacy=0x7f55ec06bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:56.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.722+0000 7f55ea59c640 1 -- 172.21.15.129:0/3296588287 >> v1:172.21.15.129:6789/0 conn(0x7f55ec0c6000 legacy=0x7f55ec164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:56.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.722+0000 7f55ea59c640 1 -- 172.21.15.129:0/3296588287 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec15a9e0 con 0x7f55ec1544b0 2024-03-23T16:00:56.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.726+0000 7f55ea59c640 1 --2- 172.21.15.129:0/3296588287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec15a2e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0097a0 tx=0x7f55dc02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e43906872987c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:56.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.730+0000 7f55e9d9b640 1 -- 172.21.15.129:0/3296588287 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc02f810 con 0x7f55ec1544b0 2024-03-23T16:00:56.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.730+0000 7f55e9d9b640 1 -- 172.21.15.129:0/3296588287 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55dc02f970 con 0x7f55ec1544b0 2024-03-23T16:00:56.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.730+0000 7f55e9d9b640 1 -- 172.21.15.129:0/3296588287 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc02fd00 con 0x7f55ec1544b0 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 -- 172.21.15.129:0/3296588287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 msgr2=0x7f55ec15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 --2- 172.21.15.129:0/3296588287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec15a2e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0097a0 tx=0x7f55dc02b9f0 comp rx=0 tx=0).stop 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 -- 172.21.15.129:0/3296588287 shutdown_connections 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 --2- 172.21.15.129:0/3296588287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec15a2e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 -- 172.21.15.129:0/3296588287 >> 172.21.15.129:0/3296588287 conn(0x7f55ec061b50 msgr2=0x7f55ec061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 -- 172.21.15.129:0/3296588287 shutdown_connections 2024-03-23T16:00:56.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.731+0000 7f55f0e84640 1 -- 172.21.15.129:0/3296588287 wait complete. 2024-03-23T16:00:56.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.733+0000 7f55f0e84640 1 Processor -- start 2024-03-23T16:00:56.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.733+0000 7f55f0e84640 1 -- start start 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.734+0000 7f55f0e84640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ec0c6000 0x7f55ec0cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.734+0000 7f55f0e84640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ec0caa00 0x7f55ec0ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.734+0000 7f55f0e84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.734+0000 7f55f0e84640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec1579c0 con 0x7f55ec1544b0 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.734+0000 7f55f0e84640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec156140 con 0x7f55ec0c6000 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.734+0000 7f55f0e84640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55ec156420 con 0x7f55ec0caa00 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.735+0000 7f55ead9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.735+0000 7f55ea59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ec0c6000 0x7f55ec0cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.735+0000 7f55ead9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52104/0 (socket says 172.21.15.129:52104) 2024-03-23T16:00:56.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.735+0000 7f55ead9d640 1 -- 172.21.15.129:0/2500815918 learned_addr learned my addr 172.21.15.129:0/2500815918 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:56.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.735+0000 7f55e3fff640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ec0caa00 0x7f55ec0ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:56.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.738+0000 7f55ead9d640 1 -- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ec0caa00 msgr2=0x7f55ec0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:56.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.738+0000 7f55ead9d640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ec0caa00 0x7f55ec0ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:56.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.738+0000 7f55ead9d640 1 -- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ec0c6000 msgr2=0x7f55ec0cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:56.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.738+0000 7f55ead9d640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ec0c6000 0x7f55ec0cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:56.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.738+0000 7f55ead9d640 1 -- 172.21.15.129:0/2500815918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc009400 con 0x7f55ec1544b0 2024-03-23T16:00:56.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.741+0000 7f55ead9d640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec157000 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f55e400c7a0 tx=0x7f55e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:56.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.744+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4003b70 con 0x7f55ec1544b0 2024-03-23T16:00:56.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.744+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e4003cf0 con 0x7f55ec1544b0 2024-03-23T16:00:56.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.744+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4003ea0 con 0x7f55ec1544b0 2024-03-23T16:00:56.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.744+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ec0d0300 con 0x7f55ec1544b0 2024-03-23T16:00:56.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.745+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ec0cfd30 con 0x7f55ec1544b0 2024-03-23T16:00:56.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.746+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ec0d0520 con 0x7f55ec1544b0 2024-03-23T16:00:57.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:56.999+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f55e400aa90 con 0x7f55ec1544b0 2024-03-23T16:00:57.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.000+0000 7f55e2ffd640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f55cc06b810 0x7f55cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:57.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.000+0000 7f55ea59c640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f55cc06b810 0x7f55cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:57.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.000+0000 7f55ea59c640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f55cc06b810 0x7f55cc06dcd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55dc036000 tx=0x7f55dc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.013+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f55e4059210 con 0x7f55ec1544b0 2024-03-23T16:00:57.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.013+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f55e40554b0 con 0x7f55ec1544b0 2024-03-23T16:00:57.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.243+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55ec075740 con 0x7f55ec1544b0 2024-03-23T16:00:57.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.273+0000 7f55e2ffd640 1 -- 172.21.15.129:0/2500815918 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f55e4058b70 con 0x7f55ec1544b0 2024-03-23T16:00:57.275 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:57.275 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:57.275 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f55cc06b810 msgr2=0x7f55cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f55cc06b810 0x7f55cc06dcd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55dc036000 tx=0x7f55dc008030 comp rx=0 tx=0).stop 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 msgr2=0x7f55ec157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec157000 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f55e400c7a0 tx=0x7f55e400cc60 comp rx=0 tx=0).stop 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 shutdown_connections 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55ec1544b0 0x7f55ec157000 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55ec0caa00 0x7f55ec0ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f55cc06b810 0x7f55cc06dcd0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 --2- 172.21.15.129:0/2500815918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55ec0c6000 0x7f55ec0cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 >> 172.21.15.129:0/2500815918 conn(0x7f55ec061b50 msgr2=0x7f55ec14a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 shutdown_connections 2024-03-23T16:00:57.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:57.275+0000 7f55f0e84640 1 -- 172.21.15.129:0/2500815918 wait complete. 2024-03-23T16:00:57.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:00:57.894 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:00:58.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:00:59.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 Processor -- start 2024-03-23T16:00:59.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 -- start start 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f10541544b0 0x7f105415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f10540caa00 0x7f105406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f10540c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f10540c5be0 con 0x7f10541544b0 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f10540cbcb0 con 0x7f10540caa00 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.998+0000 7f1059a17640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f105415b280 con 0x7f105406c0b0 2024-03-23T16:00:59.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.999+0000 7f10537fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f10540c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.999+0000 7f1052ffd640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f10541544b0 0x7f105415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.999+0000 7f10527fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f10540caa00 0x7f105406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.999+0000 7f10537fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f10540c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34294/0 (socket says 172.21.15.129:34294) 2024-03-23T16:00:59.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:58.999+0000 7f10537fe640 1 -- 172.21.15.129:0/1612919128 learned_addr learned my addr 172.21.15.129:0/1612919128 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:59.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.002+0000 7f10537fe640 1 -- 172.21.15.129:0/1612919128 >> v2:172.21.15.157:3300/0 conn(0x7f10540caa00 msgr2=0x7f105406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:59.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.002+0000 7f10537fe640 1 --2- 172.21.15.129:0/1612919128 >> v2:172.21.15.157:3300/0 conn(0x7f10540caa00 0x7f105406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.002+0000 7f10537fe640 1 -- 172.21.15.129:0/1612919128 >> v2:172.21.15.129:3300/0 conn(0x7f10541544b0 msgr2=0x7f105415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:00:59.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.002+0000 7f10537fe640 1 --2- 172.21.15.129:0/1612919128 >> v2:172.21.15.129:3300/0 conn(0x7f10541544b0 0x7f105415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.002+0000 7f10537fe640 1 -- 172.21.15.129:0/1612919128 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105415a9e0 con 0x7f105406c0b0 2024-03-23T16:00:59.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.004+0000 7f10537fe640 1 --2- 172.21.15.129:0/1612919128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f10540c6000 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f105406ce60 tx=0x7f104c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e0698aaae2b353 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:59.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.008+0000 7f1051ffb640 1 -- 172.21.15.129:0/1612919128 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c02cb00 con 0x7f105406c0b0 2024-03-23T16:00:59.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.008+0000 7f1051ffb640 1 -- 172.21.15.129:0/1612919128 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c02cc60 con 0x7f105406c0b0 2024-03-23T16:00:59.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.008+0000 7f1051ffb640 1 -- 172.21.15.129:0/1612919128 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c0343f0 con 0x7f105406c0b0 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 -- 172.21.15.129:0/1612919128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 msgr2=0x7f10540c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 --2- 172.21.15.129:0/1612919128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f10540c6000 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f105406ce60 tx=0x7f104c02bf60 comp rx=0 tx=0).stop 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 -- 172.21.15.129:0/1612919128 shutdown_connections 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 --2- 172.21.15.129:0/1612919128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f10540c6000 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 --2- 172.21.15.129:0/1612919128 >> v2:172.21.15.157:3300/0 conn(0x7f10540caa00 0x7f105406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 --2- 172.21.15.129:0/1612919128 >> v2:172.21.15.129:3300/0 conn(0x7f10541544b0 0x7f105415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 -- 172.21.15.129:0/1612919128 >> 172.21.15.129:0/1612919128 conn(0x7f1054061b50 msgr2=0x7f1054061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 -- 172.21.15.129:0/1612919128 shutdown_connections 2024-03-23T16:00:59.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.009+0000 7f1059a17640 1 -- 172.21.15.129:0/1612919128 wait complete. 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.010+0000 7f1059a17640 1 Processor -- start 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.010+0000 7f1059a17640 1 -- start start 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1059a17640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f1054154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1059a17640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540caa00 0x7f10540c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1059a17640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10541544b0 0x7f10540cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1059a17640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1054155ea0 con 0x7f10541544b0 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1059a17640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1054155970 con 0x7f10540caa00 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1059a17640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1054155bc0 con 0x7f105406c0b0 2024-03-23T16:00:59.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f1052ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f1054154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.011+0000 7f10527fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540caa00 0x7f10540c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.012+0000 7f1052ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f1054154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34308/0 (socket says 172.21.15.129:34308) 2024-03-23T16:00:59.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.012+0000 7f1052ffd640 1 -- 172.21.15.129:0/161734504 learned_addr learned my addr 172.21.15.129:0/161734504 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:00:59.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.012+0000 7f10537fe640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10541544b0 0x7f10540cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.015+0000 7f1052ffd640 1 -- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540caa00 msgr2=0x7f10540c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:59.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.015+0000 7f1052ffd640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540caa00 0x7f10540c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.015+0000 7f1052ffd640 1 -- 172.21.15.129:0/161734504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10541544b0 msgr2=0x7f10540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:59.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.015+0000 7f1052ffd640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10541544b0 0x7f10540cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.015+0000 7f1052ffd640 1 -- 172.21.15.129:0/161734504 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c009b30 con 0x7f105406c0b0 2024-03-23T16:00:59.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.017+0000 7f1052ffd640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f1054154bd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f104400e870 tx=0x7f104400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:59.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.020+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10440094f0 con 0x7f105406c0b0 2024-03-23T16:00:59.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.020+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10541f41c0 con 0x7f105406c0b0 2024-03-23T16:00:59.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.020+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10540ceb30 con 0x7f105406c0b0 2024-03-23T16:00:59.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.021+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1054154890 con 0x7f105406c0b0 2024-03-23T16:00:59.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.022+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1044005990 con 0x7f105406c0b0 2024-03-23T16:00:59.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.022+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044005d00 con 0x7f105406c0b0 2024-03-23T16:00:59.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.308+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f104400c940 con 0x7f105406c0b0 2024-03-23T16:00:59.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.308+0000 7f103bfff640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f104006b840 0x7f104006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:00:59.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.309+0000 7f10527fc640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f104006b840 0x7f104006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:00:59.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.309+0000 7f10527fc640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f104006b840 0x7f104006dd00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f105406c080 tx=0x7f1048009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:00:59.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.313+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f10440591f0 con 0x7f105406c0b0 2024-03-23T16:00:59.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.324+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1044055490 con 0x7f105406c0b0 2024-03-23T16:00:59.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.547+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f105406c490 con 0x7f105406c0b0 2024-03-23T16:00:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.568+0000 7f103bfff640 1 -- 172.21.15.129:0/161734504 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f1044021020 con 0x7f105406c0b0 2024-03-23T16:00:59.570 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:00:59.570 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:00:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:00:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f104006b840 msgr2=0x7f104006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f104006b840 0x7f104006dd00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f105406c080 tx=0x7f1048009190 comp rx=0 tx=0).stop 2024-03-23T16:00:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 msgr2=0x7f1054154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f1054154bd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f104400e870 tx=0x7f104400ed30 comp rx=0 tx=0).stop 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 shutdown_connections 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10541544b0 0x7f10540cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f104006b840 0x7f104006dd00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10540caa00 0x7f10540c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 --2- 172.21.15.129:0/161734504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f105406c0b0 0x7f1054154bd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 >> 172.21.15.129:0/161734504 conn(0x7f1054061b50 msgr2=0x7f1054061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 shutdown_connections 2024-03-23T16:00:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:00:59.570+0000 7f1059a17640 1 -- 172.21.15.129:0/161734504 wait complete. 2024-03-23T16:01:00.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:00.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:01.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:01.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.292+0000 7fabcfd4f640 1 Processor -- start 2024-03-23T16:01:01.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.293+0000 7fabcfd4f640 1 -- start start 2024-03-23T16:01:01.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.293+0000 7fabcfd4f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fabc80c8f70 0x7fabc8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:01.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.293+0000 7fabcfd4f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc8071530 0x7fabc80c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:01.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.293+0000 7fabcfd4f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc81591e0 con 0x7fabc8141630 2024-03-23T16:01:01.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.293+0000 7fabcfd4f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fabc815c740 con 0x7fabc80c8f70 2024-03-23T16:01:01.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.293+0000 7fabcfd4f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fabc8141a10 con 0x7fabc8071530 2024-03-23T16:01:01.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.294+0000 7fabcd2ea640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fabc80c8f70 0x7fabc8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:01.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.294+0000 7fabcdaeb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fabc8141630 0x7fabc815c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44936/0 (socket says 172.21.15.129:44936) 2024-03-23T16:01:01.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.294+0000 7fabcdaeb640 1 -- 172.21.15.129:0/312013835 learned_addr learned my addr 172.21.15.129:0/312013835 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:01.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.294+0000 7fabce2ec640 1 --2- 172.21.15.129:0/312013835 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc8071530 0x7fabc80c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:01.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.297+0000 7fabccae9640 1 -- 172.21.15.129:0/312013835 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3508703376 0 0) 0x7fabc4001110 con 0x7fabc8141630 2024-03-23T16:01:01.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.297+0000 7fabccae9640 1 -- 172.21.15.129:0/312013835 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb4003520 con 0x7fabc8141630 2024-03-23T16:01:01.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.297+0000 7fabcd2ea640 1 -- 172.21.15.129:0/312013835 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc8071530 msgr2=0x7fabc80c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:01.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.297+0000 7fabcd2ea640 1 --2- 172.21.15.129:0/312013835 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc8071530 0x7fabc80c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.297+0000 7fabcd2ea640 1 -- 172.21.15.129:0/312013835 >> v1:172.21.15.157:6789/0 conn(0x7fabc8141630 legacy=0x7fabc815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:01.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.297+0000 7fabcd2ea640 1 -- 172.21.15.129:0/312013835 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc815bba0 con 0x7fabc80c8f70 2024-03-23T16:01:01.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.300+0000 7fabcd2ea640 1 --2- 172.21.15.129:0/312013835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80c8f70 0x7fabc8158ca0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fabb80097a0 tx=0x7fabb802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03bac0b7a4c5fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:01.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.304+0000 7fabccae9640 1 -- 172.21.15.129:0/312013835 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb802d040 con 0x7fabc80c8f70 2024-03-23T16:01:01.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.304+0000 7fabccae9640 1 -- 172.21.15.129:0/312013835 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb8031810 con 0x7fabc80c8f70 2024-03-23T16:01:01.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.304+0000 7fabccae9640 1 -- 172.21.15.129:0/312013835 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8031ba0 con 0x7fabc80c8f70 2024-03-23T16:01:01.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 -- 172.21.15.129:0/312013835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80c8f70 msgr2=0x7fabc8158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:01.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/312013835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80c8f70 0x7fabc8158ca0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fabb80097a0 tx=0x7fabb802b8b0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 -- 172.21.15.129:0/312013835 shutdown_connections 2024-03-23T16:01:01.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/312013835 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc8071530 0x7fabc80c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/312013835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80c8f70 0x7fabc8158ca0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 -- 172.21.15.129:0/312013835 >> 172.21.15.129:0/312013835 conn(0x7fabc80c8370 msgr2=0x7fabc80c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:01.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 -- 172.21.15.129:0/312013835 shutdown_connections 2024-03-23T16:01:01.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.305+0000 7fabcfd4f640 1 -- 172.21.15.129:0/312013835 wait complete. 2024-03-23T16:01:01.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 Processor -- start 2024-03-23T16:01:01.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 -- start start 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 0x7fabc8156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc80c8f70 0x7fabc813b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc8141630 0x7fabc813bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabc813b470 con 0x7fabc8071530 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fabc81eedc0 con 0x7fabc8141630 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.307+0000 7fabcfd4f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fabc8158c20 con 0x7fabc80c8f70 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.308+0000 7fabcdaeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 0x7fabc8156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.308+0000 7fabcd2ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc80c8f70 0x7fabc813b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.308+0000 7fabce2ec640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc8141630 0x7fabc813bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.308+0000 7fabcdaeb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 0x7fabc8156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58314/0 (socket says 172.21.15.129:58314) 2024-03-23T16:01:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.308+0000 7fabcdaeb640 1 -- 172.21.15.129:0/1602040893 learned_addr learned my addr 172.21.15.129:0/1602040893 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:01.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.311+0000 7fabcdaeb640 1 -- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc80c8f70 msgr2=0x7fabc813b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:01.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.311+0000 7fabcdaeb640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc80c8f70 0x7fabc813b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.311+0000 7fabcdaeb640 1 -- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc8141630 msgr2=0x7fabc813bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:01.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.311+0000 7fabcdaeb640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc8141630 0x7fabc813bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.311+0000 7fabcdaeb640 1 -- 172.21.15.129:0/1602040893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb8009400 con 0x7fabc8071530 2024-03-23T16:01:01.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.314+0000 7fabcdaeb640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 0x7fabc8156d10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fabc400c3c0 tx=0x7fabc400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:01.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.317+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4005ce0 con 0x7fabc8071530 2024-03-23T16:01:01.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.317+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc4005e60 con 0x7fabc8071530 2024-03-23T16:01:01.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.317+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc8158e30 con 0x7fabc8071530 2024-03-23T16:01:01.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.317+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4007310 con 0x7fabc8071530 2024-03-23T16:01:01.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.318+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc813b090 con 0x7fabc8071530 2024-03-23T16:01:01.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.319+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc8075740 con 0x7fabc8071530 2024-03-23T16:01:01.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.572+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fabc4007470 con 0x7fabc8071530 2024-03-23T16:01:01.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.573+0000 7fabbe7fc640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabac06b810 0x7fabac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:01.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.573+0000 7fabcd2ea640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabac06b810 0x7fabac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:01.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.574+0000 7fabcd2ea640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabac06b810 0x7fabac06dcd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fabb8036000 tx=0x7fabb803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:01.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.587+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fabc40581f0 con 0x7fabc8071530 2024-03-23T16:01:01.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.587+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fabc4054490 con 0x7fabc8071530 2024-03-23T16:01:01.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.812+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabc81591e0 con 0x7fabc8071530 2024-03-23T16:01:01.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.844+0000 7fabbe7fc640 1 -- 172.21.15.129:0/1602040893 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fabc81591e0 con 0x7fabc8071530 2024-03-23T16:01:01.845 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:01.846 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabac06b810 msgr2=0x7fabac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabac06b810 0x7fabac06dcd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fabb8036000 tx=0x7fabb803c040 comp rx=0 tx=0).stop 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 msgr2=0x7fabc8156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 0x7fabc8156d10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fabc400c3c0 tx=0x7fabc400c880 comp rx=0 tx=0).stop 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 shutdown_connections 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabac06b810 0x7fabac06dcd0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc8141630 0x7fabc813bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fabc80c8f70 0x7fabc813b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 --2- 172.21.15.129:0/1602040893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc8071530 0x7fabc8156d10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:01.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 >> 172.21.15.129:0/1602040893 conn(0x7fabc80c8370 msgr2=0x7fabc80c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:01.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 shutdown_connections 2024-03-23T16:01:01.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:01.847+0000 7fabcfd4f640 1 -- 172.21.15.129:0/1602040893 wait complete. 2024-03-23T16:01:02.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:02.467 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:03.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.568+0000 7f251dde9640 1 Processor -- start 2024-03-23T16:01:03.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.568+0000 7f251dde9640 1 -- start start 2024-03-23T16:01:03.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.569+0000 7f251dde9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f251806c030 0x7f251806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:03.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.569+0000 7f251dde9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f251806cae0 con 0x7f25180cc080 2024-03-23T16:01:03.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.569+0000 7f251dde9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25180cabc0 con 0x7f25180cbc60 2024-03-23T16:01:03.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.569+0000 7f251dde9640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f251815e720 con 0x7f251806c030 2024-03-23T16:01:03.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.570+0000 7f2516ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f25180cbc60 0x7f251815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44940/0 (socket says 172.21.15.129:44940) 2024-03-23T16:01:03.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.570+0000 7f2516ffd640 1 -- 172.21.15.129:0/2143321814 learned_addr learned my addr 172.21.15.129:0/2143321814 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:03.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.570+0000 7f25177fe640 1 --2- 172.21.15.129:0/2143321814 >> v2:172.21.15.129:3300/0 conn(0x7f251806c030 0x7f251806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.573+0000 7f25167fc640 1 -- 172.21.15.129:0/2143321814 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828696631 0 0) 0x7f25180cabc0 con 0x7f25180cbc60 2024-03-23T16:01:03.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.573+0000 7f25167fc640 1 -- 172.21.15.129:0/2143321814 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2500003520 con 0x7f25180cbc60 2024-03-23T16:01:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.574+0000 7f25177fe640 1 -- 172.21.15.129:0/2143321814 >> v1:172.21.15.157:6789/0 conn(0x7f25180cbc60 legacy=0x7f251815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.574+0000 7f25177fe640 1 -- 172.21.15.129:0/2143321814 >> v1:172.21.15.129:6789/0 conn(0x7f25180cc080 legacy=0x7f251815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.574+0000 7f25177fe640 1 -- 172.21.15.129:0/2143321814 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25180caea0 con 0x7f251806c030 2024-03-23T16:01:03.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.578+0000 7f25177fe640 1 --2- 172.21.15.129:0/2143321814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251806c030 0x7f251806c410 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f25080097a0 tx=0x7f250802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9634534431e57f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.583+0000 7f25167fc640 1 -- 172.21.15.129:0/2143321814 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2508030610 con 0x7f251806c030 2024-03-23T16:01:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.583+0000 7f25167fc640 1 -- 172.21.15.129:0/2143321814 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2508030770 con 0x7f251806c030 2024-03-23T16:01:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.583+0000 7f25167fc640 1 -- 172.21.15.129:0/2143321814 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2508030b00 con 0x7f251806c030 2024-03-23T16:01:03.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 -- 172.21.15.129:0/2143321814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251806c030 msgr2=0x7f251806c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:03.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 --2- 172.21.15.129:0/2143321814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251806c030 0x7f251806c410 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f25080097a0 tx=0x7f250802b960 comp rx=0 tx=0).stop 2024-03-23T16:01:03.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 -- 172.21.15.129:0/2143321814 shutdown_connections 2024-03-23T16:01:03.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 --2- 172.21.15.129:0/2143321814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251806c030 0x7f251806c410 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:03.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 -- 172.21.15.129:0/2143321814 >> 172.21.15.129:0/2143321814 conn(0x7f25180cfb20 msgr2=0x7f25180cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:03.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 -- 172.21.15.129:0/2143321814 shutdown_connections 2024-03-23T16:01:03.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.584+0000 7f251dde9640 1 -- 172.21.15.129:0/2143321814 wait complete. 2024-03-23T16:01:03.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.586+0000 7f251dde9640 1 Processor -- start 2024-03-23T16:01:03.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.586+0000 7f251dde9640 1 -- start start 2024-03-23T16:01:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.587+0000 7f251dde9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251806c030 0x7f25181599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.587+0000 7f251dde9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25180cbc60 0x7f2518156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.587+0000 7f251dde9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 0x7f2518151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:03.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.587+0000 7f251dde9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25181571b0 con 0x7f25180cc080 2024-03-23T16:01:03.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.587+0000 7f251dde9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2518157d70 con 0x7f251806c030 2024-03-23T16:01:03.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.587+0000 7f251dde9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f25181f4380 con 0x7f25180cbc60 2024-03-23T16:01:03.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.588+0000 7f2517fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 0x7f2518151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:03.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.588+0000 7f25177fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251806c030 0x7f25181599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.588+0000 7f2516ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25180cbc60 0x7f2518156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.588+0000 7f2517fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 0x7f2518151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58332/0 (socket says 172.21.15.129:58332) 2024-03-23T16:01:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.588+0000 7f2517fff640 1 -- 172.21.15.129:0/1172259417 learned_addr learned my addr 172.21.15.129:0/1172259417 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.593+0000 7f2517fff640 1 -- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25180cbc60 msgr2=0x7f2518156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.593+0000 7f2517fff640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25180cbc60 0x7f2518156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.593+0000 7f2517fff640 1 -- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251806c030 msgr2=0x7f25181599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.593+0000 7f2517fff640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251806c030 0x7f25181599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.593+0000 7f2517fff640 1 -- 172.21.15.129:0/1172259417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2508009400 con 0x7f25180cc080 2024-03-23T16:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.596+0000 7f2517fff640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 0x7f2518151040 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f251000c6b0 tx=0x7f251000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:03.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.600+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510003b70 con 0x7f25180cc080 2024-03-23T16:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.601+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2510003cf0 con 0x7f25180cc080 2024-03-23T16:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.601+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510003ea0 con 0x7f25180cc080 2024-03-23T16:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.601+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25181f3dc0 con 0x7f25180cc080 2024-03-23T16:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.601+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2518155b90 con 0x7f25180cc080 2024-03-23T16:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.602+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2518075740 con 0x7f25180cc080 2024-03-23T16:01:03.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.632+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2510005310 con 0x7f25180cc080 2024-03-23T16:01:03.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.632+0000 7f24fffff640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250406b860 0x7f250406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:03.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.633+0000 7f25177fe640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250406b860 0x7f250406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:03.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.633+0000 7f25177fe640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250406b860 0x7f250406dd20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2508036000 tx=0x7f2508008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:03.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.897+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2510059060 con 0x7f25180cc080 2024-03-23T16:01:03.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:03.897+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2510055300 con 0x7f25180cc080 2024-03-23T16:01:04.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.133+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25180d0a30 con 0x7f25180cc080 2024-03-23T16:01:04.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.148+0000 7f24fffff640 1 -- 172.21.15.129:0/1172259417 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f25100589c0 con 0x7f25180cc080 2024-03-23T16:01:04.150 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:04.150 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:04.151 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250406b860 msgr2=0x7f250406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250406b860 0x7f250406dd20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2508036000 tx=0x7f2508008030 comp rx=0 tx=0).stop 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 msgr2=0x7f2518151040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 0x7f2518151040 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f251000c6b0 tx=0x7f251000cb70 comp rx=0 tx=0).stop 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 shutdown_connections 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25180cc080 0x7f2518151040 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25180cbc60 0x7f2518156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250406b860 0x7f250406dd20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 --2- 172.21.15.129:0/1172259417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251806c030 0x7f25181599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:04.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 >> 172.21.15.129:0/1172259417 conn(0x7f25180cfb20 msgr2=0x7f251815cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:04.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 shutdown_connections 2024-03-23T16:01:04.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:04.150+0000 7f251dde9640 1 -- 172.21.15.129:0/1172259417 wait complete. 2024-03-23T16:01:04.769 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:04.769 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:05.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:05.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.873+0000 7f73ddcda640 1 Processor -- start 2024-03-23T16:01:05.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.873+0000 7f73ddcda640 1 -- start start 2024-03-23T16:01:05.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.874+0000 7f73ddcda640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f73d81544b0 0x7f73d815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:05.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.874+0000 7f73ddcda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d80caa00 0x7f73d806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:05.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.874+0000 7f73ddcda640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d80cbcb0 con 0x7f73d806c0b0 2024-03-23T16:01:05.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.874+0000 7f73ddcda640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f73d80c5be0 con 0x7f73d81544b0 2024-03-23T16:01:05.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.874+0000 7f73ddcda640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73d815b280 con 0x7f73d80caa00 2024-03-23T16:01:05.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.875+0000 7f73d6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d80caa00 0x7f73d806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:05.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.875+0000 7f73d77fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f73d81544b0 0x7f73d815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:05.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.875+0000 7f73d7fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f73d806c0b0 0x7f73d80c6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:37914/0 (socket says 172.21.15.129:37914) 2024-03-23T16:01:05.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.875+0000 7f73d7fff640 1 -- 172.21.15.129:0/1335455390 learned_addr learned my addr 172.21.15.129:0/1335455390 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.879+0000 7f73d6ffd640 1 -- 172.21.15.129:0/1335455390 >> v2:172.21.15.129:3300/0 conn(0x7f73d81544b0 msgr2=0x7f73d815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.879+0000 7f73d6ffd640 1 --2- 172.21.15.129:0/1335455390 >> v2:172.21.15.129:3300/0 conn(0x7f73d81544b0 0x7f73d815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.879+0000 7f73d6ffd640 1 -- 172.21.15.129:0/1335455390 >> v1:172.21.15.129:6789/0 conn(0x7f73d806c0b0 legacy=0x7f73d80c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.879+0000 7f73d6ffd640 1 -- 172.21.15.129:0/1335455390 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d815a9e0 con 0x7f73d80caa00 2024-03-23T16:01:05.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.881+0000 7f73d6ffd640 1 --2- 172.21.15.129:0/1335455390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d80caa00 0x7f73d806bb70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f73c00097a0 tx=0x7f73c002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f53512930d9d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:05.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.885+0000 7f73d67fc640 1 -- 172.21.15.129:0/1335455390 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0030610 con 0x7f73d80caa00 2024-03-23T16:01:05.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.885+0000 7f73d67fc640 1 -- 172.21.15.129:0/1335455390 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73c0030770 con 0x7f73d80caa00 2024-03-23T16:01:05.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.885+0000 7f73d67fc640 1 -- 172.21.15.129:0/1335455390 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0030b00 con 0x7f73d80caa00 2024-03-23T16:01:05.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.886+0000 7f73ddcda640 1 -- 172.21.15.129:0/1335455390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d80caa00 msgr2=0x7f73d806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.886+0000 7f73ddcda640 1 --2- 172.21.15.129:0/1335455390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d80caa00 0x7f73d806bb70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f73c00097a0 tx=0x7f73c002b960 comp rx=0 tx=0).stop 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.887+0000 7f73ddcda640 1 -- 172.21.15.129:0/1335455390 shutdown_connections 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.887+0000 7f73ddcda640 1 --2- 172.21.15.129:0/1335455390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d80caa00 0x7f73d806bb70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.887+0000 7f73ddcda640 1 --2- 172.21.15.129:0/1335455390 >> v2:172.21.15.129:3300/0 conn(0x7f73d81544b0 0x7f73d815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.887+0000 7f73ddcda640 1 -- 172.21.15.129:0/1335455390 >> 172.21.15.129:0/1335455390 conn(0x7f73d8061b50 msgr2=0x7f73d8061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.887+0000 7f73ddcda640 1 -- 172.21.15.129:0/1335455390 shutdown_connections 2024-03-23T16:01:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.887+0000 7f73ddcda640 1 -- 172.21.15.129:0/1335455390 wait complete. 2024-03-23T16:01:05.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.888+0000 7f73ddcda640 1 Processor -- start 2024-03-23T16:01:05.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.889+0000 7f73ddcda640 1 -- start start 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73ddcda640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 0x7f73d80cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73ddcda640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d80caa00 0x7f73d8156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73ddcda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d81544b0 0x7f73d81569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73ddcda640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73d8156040 con 0x7f73d806c0b0 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73ddcda640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73d8148650 con 0x7f73d80caa00 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73ddcda640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73d81489d0 con 0x7f73d81544b0 2024-03-23T16:01:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73d77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 0x7f73d80cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:05.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.890+0000 7f73d7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d81544b0 0x7f73d81569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:05.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.891+0000 7f73d77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 0x7f73d80cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58350/0 (socket says 172.21.15.129:58350) 2024-03-23T16:01:05.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.891+0000 7f73d77fe640 1 -- 172.21.15.129:0/4185030099 learned_addr learned my addr 172.21.15.129:0/4185030099 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:05.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.891+0000 7f73d6ffd640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d80caa00 0x7f73d8156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:05.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.894+0000 7f73d77fe640 1 -- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d81544b0 msgr2=0x7f73d81569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:05.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.894+0000 7f73d77fe640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d81544b0 0x7f73d81569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:05.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.894+0000 7f73d77fe640 1 -- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d80caa00 msgr2=0x7f73d8156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:05.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.894+0000 7f73d77fe640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d80caa00 0x7f73d8156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:05.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.894+0000 7f73d77fe640 1 -- 172.21.15.129:0/4185030099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c0009400 con 0x7f73d806c0b0 2024-03-23T16:01:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.896+0000 7f73d77fe640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 0x7f73d80cfa40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f73d000e870 tx=0x7f73d000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.900+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d00094f0 con 0x7f73d806c0b0 2024-03-23T16:01:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.900+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d000c940 con 0x7f73d806c0b0 2024-03-23T16:01:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.900+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d000ccf0 con 0x7f73d806c0b0 2024-03-23T16:01:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.900+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d8145c10 con 0x7f73d806c0b0 2024-03-23T16:01:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.900+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d80d0580 con 0x7f73d806c0b0 2024-03-23T16:01:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:05.902+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d80b5780 con 0x7f73d806c0b0 2024-03-23T16:01:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.153+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f73d0013470 con 0x7f73d806c0b0 2024-03-23T16:01:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.153+0000 7f73cffff640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b806b890 0x7f73b806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.154+0000 7f73d6ffd640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b806b890 0x7f73b806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:06.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.154+0000 7f73d6ffd640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b806b890 0x7f73b806dd50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f73c0036000 tx=0x7f73c0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:06.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.170+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f73d0014e40 con 0x7f73d806c0b0 2024-03-23T16:01:06.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.170+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73d00554a0 con 0x7f73d806c0b0 2024-03-23T16:01:06.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.389+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73d8075740 con 0x7f73d806c0b0 2024-03-23T16:01:06.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.404+0000 7f73cffff640 1 -- 172.21.15.129:0/4185030099 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f73d0021090 con 0x7f73d806c0b0 2024-03-23T16:01:06.407 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:06.407 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:06.407 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.407+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b806b890 msgr2=0x7f73b806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.407+0000 7f73ddcda640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b806b890 0x7f73b806dd50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f73c0036000 tx=0x7f73c0008030 comp rx=0 tx=0).stop 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.407+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 msgr2=0x7f73d80cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.407+0000 7f73ddcda640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 0x7f73d80cfa40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f73d000e870 tx=0x7f73d000ed30 comp rx=0 tx=0).stop 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 shutdown_connections 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73d81544b0 0x7f73d81569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f73b806b890 0x7f73b806dd50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d80caa00 0x7f73d8156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:06.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 --2- 172.21.15.129:0/4185030099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d806c0b0 0x7f73d80cfa40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:06.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 >> 172.21.15.129:0/4185030099 conn(0x7f73d8061b50 msgr2=0x7f73d814a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:06.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 shutdown_connections 2024-03-23T16:01:06.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:06.408+0000 7f73ddcda640 1 -- 172.21.15.129:0/4185030099 wait complete. 2024-03-23T16:01:07.026 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:07.026 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:08.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:08.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.126+0000 7efdb4a84640 1 Processor -- start 2024-03-23T16:01:08.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.126+0000 7efdb4a84640 1 -- start start 2024-03-23T16:01:08.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.128+0000 7efdb4a84640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efdb006c410 0x7efdb015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:08.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.128+0000 7efdb4a84640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb006c830 con 0x7efdb015de40 2024-03-23T16:01:08.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.128+0000 7efdb4a84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb00cc370 con 0x7efdb006c030 2024-03-23T16:01:08.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.128+0000 7efdb4a84640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efdb006cae0 con 0x7efdb006c410 2024-03-23T16:01:08.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.128+0000 7efdaed9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7efdb015de40 0x7efdb0164c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:37922/0 (socket says 172.21.15.129:37922) 2024-03-23T16:01:08.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.128+0000 7efdaed9d640 1 -- 172.21.15.129:0/893494008 learned_addr learned my addr 172.21.15.129:0/893494008 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:08.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.129+0000 7efdadd9b640 1 --2- 172.21.15.129:0/893494008 >> v2:172.21.15.157:3300/0 conn(0x7efdb006c410 0x7efdb015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:08.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.131+0000 7efdad59a640 1 -- 172.21.15.129:0/893494008 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990879321 0 0) 0x7efdb006c830 con 0x7efdb015de40 2024-03-23T16:01:08.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.131+0000 7efdad59a640 1 -- 172.21.15.129:0/893494008 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd98003520 con 0x7efdb015de40 2024-03-23T16:01:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.133+0000 7efdadd9b640 1 -- 172.21.15.129:0/893494008 >> v1:172.21.15.157:6789/0 conn(0x7efdb006c030 legacy=0x7efdb00cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.133+0000 7efdadd9b640 1 -- 172.21.15.129:0/893494008 >> v1:172.21.15.129:6789/0 conn(0x7efdb015de40 legacy=0x7efdb0164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.133+0000 7efdadd9b640 1 -- 172.21.15.129:0/893494008 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb00cabc0 con 0x7efdb006c410 2024-03-23T16:01:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.140+0000 7efdadd9b640 1 --2- 172.21.15.129:0/893494008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb015d900 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efda40097a0 tx=0x7efda402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e94fbea13fa324f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:08.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.144+0000 7efdad59a640 1 -- 172.21.15.129:0/893494008 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4030610 con 0x7efdb006c410 2024-03-23T16:01:08.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.144+0000 7efdad59a640 1 -- 172.21.15.129:0/893494008 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efda4030770 con 0x7efdb006c410 2024-03-23T16:01:08.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.144+0000 7efdad59a640 1 -- 172.21.15.129:0/893494008 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4030b00 con 0x7efdb006c410 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 -- 172.21.15.129:0/893494008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 msgr2=0x7efdb015d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 --2- 172.21.15.129:0/893494008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb015d900 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efda40097a0 tx=0x7efda402b960 comp rx=0 tx=0).stop 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 -- 172.21.15.129:0/893494008 shutdown_connections 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 --2- 172.21.15.129:0/893494008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb015d900 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 -- 172.21.15.129:0/893494008 >> 172.21.15.129:0/893494008 conn(0x7efdb00cfb20 msgr2=0x7efdb00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 -- 172.21.15.129:0/893494008 shutdown_connections 2024-03-23T16:01:08.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.145+0000 7efdb4a84640 1 -- 172.21.15.129:0/893494008 wait complete. 2024-03-23T16:01:08.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.147+0000 7efdb4a84640 1 Processor -- start 2024-03-23T16:01:08.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.147+0000 7efdb4a84640 1 -- start start 2024-03-23T16:01:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.148+0000 7efdb4a84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 0x7efdb00c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.148+0000 7efdb4a84640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb0157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.148+0000 7efdb4a84640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb015de40 0x7efdb00c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.148+0000 7efdb4a84640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efdb00c4c70 con 0x7efdb006c030 2024-03-23T16:01:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.148+0000 7efdb4a84640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efdb00c5a20 con 0x7efdb006c410 2024-03-23T16:01:08.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.148+0000 7efdb4a84640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efdb01eff60 con 0x7efdb015de40 2024-03-23T16:01:08.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.149+0000 7efdae59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 0x7efdb00c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:08.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.149+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb0157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:08.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.149+0000 7efdaed9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb015de40 0x7efdb00c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:08.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.149+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb0157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:46030/0 (socket says 172.21.15.129:46030) 2024-03-23T16:01:08.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.149+0000 7efdadd9b640 1 -- 172.21.15.129:0/607654030 learned_addr learned my addr 172.21.15.129:0/607654030 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:08.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.149+0000 7efdae59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 0x7efdb00c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58354/0 (socket says 172.21.15.129:58354) 2024-03-23T16:01:08.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.152+0000 7efdae59c640 1 -- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb015de40 msgr2=0x7efdb00c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.152+0000 7efdae59c640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb015de40 0x7efdb00c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.152+0000 7efdae59c640 1 -- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 msgr2=0x7efdb0157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.152+0000 7efdae59c640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb0157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.152+0000 7efdae59c640 1 -- 172.21.15.129:0/607654030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda4009400 con 0x7efdb006c030 2024-03-23T16:01:08.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.156+0000 7efdae59c640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 0x7efdb00c54e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7efda000c390 tx=0x7efda000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:08.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.163+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda00052d0 con 0x7efdb006c030 2024-03-23T16:01:08.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.163+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efda0005430 con 0x7efdb006c030 2024-03-23T16:01:08.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.163+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb01f0200 con 0x7efdb006c030 2024-03-23T16:01:08.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.163+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda0003830 con 0x7efdb006c030 2024-03-23T16:01:08.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.163+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb01f07c0 con 0x7efdb006c030 2024-03-23T16:01:08.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.164+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb0154ac0 con 0x7efdb006c030 2024-03-23T16:01:08.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.185+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efda0003990 con 0x7efdb006c030 2024-03-23T16:01:08.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.185+0000 7efd9effd640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd9006b810 0x7efd9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:08.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.185+0000 7efdadd9b640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd9006b810 0x7efd9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:08.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.186+0000 7efdadd9b640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd9006b810 0x7efd9006dcd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efdb01f4480 tx=0x7efda4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:08.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.189+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efda0058230 con 0x7efdb006c030 2024-03-23T16:01:08.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.435+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efda00544d0 con 0x7efdb006c030 2024-03-23T16:01:08.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.684+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdb0075740 con 0x7efdb006c030 2024-03-23T16:01:08.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.705+0000 7efd9effd640 1 -- 172.21.15.129:0/607654030 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7efda0017020 con 0x7efdb006c030 2024-03-23T16:01:08.707 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:08.707 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:08.707 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd9006b810 msgr2=0x7efd9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd9006b810 0x7efd9006dcd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efdb01f4480 tx=0x7efda4008030 comp rx=0 tx=0).stop 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 msgr2=0x7efdb00c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 0x7efdb00c54e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7efda000c390 tx=0x7efda000c850 comp rx=0 tx=0).stop 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 shutdown_connections 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb015de40 0x7efdb00c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efd9006b810 0x7efd9006dcd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb006c410 0x7efdb0157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 --2- 172.21.15.129:0/607654030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdb006c030 0x7efdb00c54e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:08.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 >> 172.21.15.129:0/607654030 conn(0x7efdb00cfb20 msgr2=0x7efdb00c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:08.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 shutdown_connections 2024-03-23T16:01:08.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:08.708+0000 7efdb4a84640 1 -- 172.21.15.129:0/607654030 wait complete. 2024-03-23T16:01:09.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:01:09.247 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:01:09.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:09.325 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:10.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:10.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.428+0000 7fb7c2a15640 1 Processor -- start 2024-03-23T16:01:10.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.428+0000 7fb7c2a15640 1 -- start start 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.428+0000 7fb7c2a15640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb7bc0ccd60 0x7fb7bc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.428+0000 7fb7c2a15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7bc15b290 con 0x7fb7bc0ceb00 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.428+0000 7fb7c2a15640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7bc0ce800 con 0x7fb7bc0cef40 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.428+0000 7fb7c2a15640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb7bc154a70 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.429+0000 7fb7c0fb2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb7bc0cef40 0x7fb7bc15b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:44954/0 (socket says 172.21.15.129:44954) 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.429+0000 7fb7c0fb2640 1 -- 172.21.15.129:0/3188029733 learned_addr learned my addr 172.21.15.129:0/3188029733 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:10.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.429+0000 7fb7bbfff640 1 --2- 172.21.15.129:0/3188029733 >> v2:172.21.15.129:3300/0 conn(0x7fb7bc0ccd60 0x7fb7bc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.433+0000 7fb7bbfff640 1 -- 172.21.15.129:0/3188029733 >> v1:172.21.15.157:6789/0 conn(0x7fb7bc0cef40 legacy=0x7fb7bc15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.433+0000 7fb7bbfff640 1 -- 172.21.15.129:0/3188029733 >> v1:172.21.15.129:6789/0 conn(0x7fb7bc0ceb00 legacy=0x7fb7bc06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.433+0000 7fb7bbfff640 1 -- 172.21.15.129:0/3188029733 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc0cd950 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.438+0000 7fb7bbfff640 1 --2- 172.21.15.129:0/3188029733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc15ad50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac0097a0 tx=0x7fb7ac02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f870dfcb933eab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7baffd640 1 -- 172.21.15.129:0/3188029733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac02f810 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7baffd640 1 -- 172.21.15.129:0/3188029733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ac02f970 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7baffd640 1 -- 172.21.15.129:0/3188029733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac02fd00 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 -- 172.21.15.129:0/3188029733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 msgr2=0x7fb7bc15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/3188029733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc15ad50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac0097a0 tx=0x7fb7ac02b9f0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 -- 172.21.15.129:0/3188029733 shutdown_connections 2024-03-23T16:01:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/3188029733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc15ad50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 -- 172.21.15.129:0/3188029733 >> 172.21.15.129:0/3188029733 conn(0x7fb7bc0b1840 msgr2=0x7fb7bc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 -- 172.21.15.129:0/3188029733 shutdown_connections 2024-03-23T16:01:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.442+0000 7fb7c2a15640 1 -- 172.21.15.129:0/3188029733 wait complete. 2024-03-23T16:01:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.444+0000 7fb7c2a15640 1 Processor -- start 2024-03-23T16:01:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.444+0000 7fb7c2a15640 1 -- start start 2024-03-23T16:01:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7c2a15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc1505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7c2a15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7bc0ceb00 0x7fb7bc0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7c2a15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7bc0cef40 0x7fb7bc0b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7c2a15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc14ac00 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7c2a15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc15c920 con 0x7fb7bc0ceb00 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7c2a15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7bc1e4710 con 0x7fb7bc0cef40 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc1505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.445+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7bc0ceb00 0x7fb7bc0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.446+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc1505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58366/0 (socket says 172.21.15.129:58366) 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.446+0000 7fb7bbfff640 1 -- 172.21.15.129:0/1129508059 learned_addr learned my addr 172.21.15.129:0/1129508059 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.446+0000 7fb7c0fb2640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7bc0cef40 0x7fb7bc0b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.449+0000 7fb7bbfff640 1 -- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7bc0cef40 msgr2=0x7fb7bc0b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.449+0000 7fb7bbfff640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7bc0cef40 0x7fb7bc0b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.449+0000 7fb7bbfff640 1 -- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7bc0ceb00 msgr2=0x7fb7bc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.449+0000 7fb7bbfff640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7bc0ceb00 0x7fb7bc0b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.449+0000 7fb7bbfff640 1 -- 172.21.15.129:0/1129508059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ac009400 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.451+0000 7fb7bbfff640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc1505e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009890 tx=0x7fb7ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.455+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac03c040 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.455+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ac0379f0 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.455+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac037da0 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.455+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc15c640 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.455+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc15d5c0 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.456+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc04f530 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.712+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb7ac0403e0 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.712+0000 7fb7b8ff9640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7a806b840 0x7fb7a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.712+0000 7fb7bb7fe640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7a806b840 0x7fb7a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.713+0000 7fb7bb7fe640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7a806b840 0x7fb7a806dd00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc15e360 tx=0x7fb7b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.727+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb7ac07a980 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.728+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb7ac076c20 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.977+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7bc0686f0 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.989+0000 7fb7b8ff9640 1 -- 172.21.15.129:0/1129508059 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb7ac046070 con 0x7fb7bc0ccd60 2024-03-23T16:01:10.991 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:10.992 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:10.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7a806b840 msgr2=0x7fb7a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7a806b840 0x7fb7a806dd00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc15e360 tx=0x7fb7b0009040 comp rx=0 tx=0).stop 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 msgr2=0x7fb7bc1505e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc1505e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009890 tx=0x7fb7ac004730 comp rx=0 tx=0).stop 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 shutdown_connections 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7bc0cef40 0x7fb7bc0b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb7a806b840 0x7fb7a806dd00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7bc0ceb00 0x7fb7bc0b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 --2- 172.21.15.129:0/1129508059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7bc0ccd60 0x7fb7bc1505e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 >> 172.21.15.129:0/1129508059 conn(0x7fb7bc0b1840 msgr2=0x7fb7bc1e6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 shutdown_connections 2024-03-23T16:01:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:10.992+0000 7fb7c2a15640 1 -- 172.21.15.129:0/1129508059 wait complete. 2024-03-23T16:01:11.610 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:11.610 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:12.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:12.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.712+0000 7fb948223640 1 Processor -- start 2024-03-23T16:01:12.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.712+0000 7fb948223640 1 -- start start 2024-03-23T16:01:12.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.714+0000 7fb948223640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb94006c030 0x7fb94006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:12.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.714+0000 7fb948223640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb9400cbc60 0x7fb94015d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:12.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.714+0000 7fb948223640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb94006cae0 con 0x7fb9400cc080 2024-03-23T16:01:12.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.714+0000 7fb948223640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb9400cabc0 con 0x7fb94006c030 2024-03-23T16:01:12.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.714+0000 7fb948223640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb9400caea0 con 0x7fb9400cbc60 2024-03-23T16:01:12.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.714+0000 7fb9457be640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb9400cbc60 0x7fb94015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:12.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.715+0000 7fb9467c0640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb9400cc080 0x7fb94015de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48020/0 (socket says 172.21.15.129:48020) 2024-03-23T16:01:12.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.715+0000 7fb9467c0640 1 -- 172.21.15.129:0/1446055989 learned_addr learned my addr 172.21.15.129:0/1446055989 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:12.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.715+0000 7fb945fbf640 1 --2- 172.21.15.129:0/1446055989 >> v2:172.21.15.129:3300/0 conn(0x7fb94006c030 0x7fb94006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:12.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.717+0000 7fb9457be640 1 -- 172.21.15.129:0/1446055989 >> v2:172.21.15.129:3300/0 conn(0x7fb94006c030 msgr2=0x7fb94006c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:12.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.717+0000 7fb9457be640 1 --2- 172.21.15.129:0/1446055989 >> v2:172.21.15.129:3300/0 conn(0x7fb94006c030 0x7fb94006c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:12.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.717+0000 7fb9457be640 1 -- 172.21.15.129:0/1446055989 >> v1:172.21.15.129:6789/0 conn(0x7fb9400cc080 legacy=0x7fb94015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:12.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.717+0000 7fb9457be640 1 -- 172.21.15.129:0/1446055989 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9400cb180 con 0x7fb9400cbc60 2024-03-23T16:01:12.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.720+0000 7fb9457be640 1 --2- 172.21.15.129:0/1446055989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb94015d900 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb9400cd660 tx=0x7fb93002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ced4646e78e5d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:12.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.725+0000 7fb944fbd640 1 -- 172.21.15.129:0/1446055989 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930031830 con 0x7fb9400cbc60 2024-03-23T16:01:12.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.725+0000 7fb944fbd640 1 -- 172.21.15.129:0/1446055989 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb930031990 con 0x7fb9400cbc60 2024-03-23T16:01:12.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.725+0000 7fb944fbd640 1 -- 172.21.15.129:0/1446055989 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930031d20 con 0x7fb9400cbc60 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.725+0000 7fb948223640 1 -- 172.21.15.129:0/1446055989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 msgr2=0x7fb94015d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.725+0000 7fb948223640 1 --2- 172.21.15.129:0/1446055989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb94015d900 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb9400cd660 tx=0x7fb93002b8d0 comp rx=0 tx=0).stop 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.726+0000 7fb948223640 1 -- 172.21.15.129:0/1446055989 shutdown_connections 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.726+0000 7fb948223640 1 --2- 172.21.15.129:0/1446055989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb94015d900 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.726+0000 7fb948223640 1 --2- 172.21.15.129:0/1446055989 >> v2:172.21.15.129:3300/0 conn(0x7fb94006c030 0x7fb94006c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.726+0000 7fb948223640 1 -- 172.21.15.129:0/1446055989 >> 172.21.15.129:0/1446055989 conn(0x7fb9400cfb20 msgr2=0x7fb9400cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.726+0000 7fb948223640 1 -- 172.21.15.129:0/1446055989 shutdown_connections 2024-03-23T16:01:12.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.726+0000 7fb948223640 1 -- 172.21.15.129:0/1446055989 wait complete. 2024-03-23T16:01:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.727+0000 7fb948223640 1 Processor -- start 2024-03-23T16:01:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.728+0000 7fb948223640 1 -- start start 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb948223640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb94006c030 0x7fb9400b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb948223640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb9401efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb948223640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9400cc080 0x7fb9401f0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb948223640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb940156b20 con 0x7fb9400cc080 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb948223640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401f3170 con 0x7fb9400cbc60 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb948223640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb940157d10 con 0x7fb94006c030 2024-03-23T16:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb9467c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9400cc080 0x7fb9401f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:12.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb9467c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9400cc080 0x7fb9401f0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60878/0 (socket says 172.21.15.129:60878) 2024-03-23T16:01:12.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb9457be640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb9401efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:12.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.729+0000 7fb9467c0640 1 -- 172.21.15.129:0/2509858381 learned_addr learned my addr 172.21.15.129:0/2509858381 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:12.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.730+0000 7fb945fbf640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb94006c030 0x7fb9400b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.733+0000 7fb9457be640 1 -- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb94006c030 msgr2=0x7fb9400b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.733+0000 7fb9457be640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb94006c030 0x7fb9400b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.733+0000 7fb9457be640 1 -- 172.21.15.129:0/2509858381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9400cc080 msgr2=0x7fb9401f0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.733+0000 7fb9457be640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9400cc080 0x7fb9401f0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.733+0000 7fb9457be640 1 -- 172.21.15.129:0/2509858381 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb930009400 con 0x7fb9400cbc60 2024-03-23T16:01:12.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.736+0000 7fb9457be640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb9401efc00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb930033000 tx=0x7fb9300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:12.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.740+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93003ba70 con 0x7fb9400cbc60 2024-03-23T16:01:12.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.740+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb93003bbf0 con 0x7fb9400cbc60 2024-03-23T16:01:12.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.740+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9300453f0 con 0x7fb9400cbc60 2024-03-23T16:01:12.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.740+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9401502e0 con 0x7fb9400cbc60 2024-03-23T16:01:12.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.741+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9401f36a0 con 0x7fb9400cbc60 2024-03-23T16:01:12.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:12.742+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb940154aa0 con 0x7fb9400cbc60 2024-03-23T16:01:13.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.032+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb93004e450 con 0x7fb9400cbc60 2024-03-23T16:01:13.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.033+0000 7fb9367fc640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb92406b860 0x7fb92406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:13.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.033+0000 7fb945fbf640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb92406b860 0x7fb92406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:13.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.033+0000 7fb945fbf640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb92406b860 0x7fb92406dd20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb93c006fd0 tx=0x7fb93c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.039+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb93007aff0 con 0x7fb9400cbc60 2024-03-23T16:01:13.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.058+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb930077290 con 0x7fb9400cbc60 2024-03-23T16:01:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.321+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb940075740 con 0x7fb9400cbc60 2024-03-23T16:01:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.346+0000 7fb9367fc640 1 -- 172.21.15.129:0/2509858381 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fb930035030 con 0x7fb9400cbc60 2024-03-23T16:01:13.348 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:13.348 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:13.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb92406b860 msgr2=0x7fb92406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:13.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb92406b860 0x7fb92406dd20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb93c006fd0 tx=0x7fb93c008040 comp rx=0 tx=0).stop 2024-03-23T16:01:13.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 msgr2=0x7fb9401efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:13.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb9401efc00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb930033000 tx=0x7fb9300046c0 comp rx=0 tx=0).stop 2024-03-23T16:01:13.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 shutdown_connections 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9400cc080 0x7fb9401f0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb92406b860 0x7fb92406dd20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9400cbc60 0x7fb9401efc00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 --2- 172.21.15.129:0/2509858381 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb94006c030 0x7fb9400b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 >> 172.21.15.129:0/2509858381 conn(0x7fb9400cfb20 msgr2=0x7fb940158d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 shutdown_connections 2024-03-23T16:01:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:13.348+0000 7fb948223640 1 -- 172.21.15.129:0/2509858381 wait complete. 2024-03-23T16:01:13.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:13.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:14.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:15.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.069+0000 7f46df615640 1 Processor -- start 2024-03-23T16:01:15.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.069+0000 7f46df615640 1 -- start start 2024-03-23T16:01:15.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46df615640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d815d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:15.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46df615640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f46d815de40 0x7f46d815e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:15.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46df615640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d806c830 con 0x7f46d806c030 2024-03-23T16:01:15.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46df615640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f46d80cc370 con 0x7f46d815de40 2024-03-23T16:01:15.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46df615640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f46d806cae0 con 0x7f46d806c410 2024-03-23T16:01:15.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46ddbb2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f46d815de40 0x7f46d815e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:15.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46dcbb0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:15.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.070+0000 7f46ddbb2640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f46d815de40 0x7f46d815e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60886/0 (socket says 172.21.15.129:60886) 2024-03-23T16:01:15.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.071+0000 7f46ddbb2640 1 -- 172.21.15.129:0/797774237 learned_addr learned my addr 172.21.15.129:0/797774237 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:15.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.073+0000 7f46ddbb2640 1 -- 172.21.15.129:0/797774237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 msgr2=0x7f46d815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:15.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.073+0000 7f46ddbb2640 1 --2- 172.21.15.129:0/797774237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.073+0000 7f46ddbb2640 1 -- 172.21.15.129:0/797774237 >> v1:172.21.15.157:6789/0 conn(0x7f46d806c030 legacy=0x7f46d80cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:15.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.073+0000 7f46ddbb2640 1 -- 172.21.15.129:0/797774237 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d80cabc0 con 0x7f46d815de40 2024-03-23T16:01:15.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.075+0000 7f46ddbb2640 1 --2- 172.21.15.129:0/797774237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d815e2d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e50 tx=0x7f46d002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0c5e1d25beb79d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:15.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.078+0000 7f46cffff640 1 -- 172.21.15.129:0/797774237 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d002cb00 con 0x7f46d815de40 2024-03-23T16:01:15.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.078+0000 7f46cffff640 1 -- 172.21.15.129:0/797774237 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d002cc60 con 0x7f46d815de40 2024-03-23T16:01:15.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.078+0000 7f46cffff640 1 -- 172.21.15.129:0/797774237 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d00343f0 con 0x7f46d815de40 2024-03-23T16:01:15.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.079+0000 7f46df615640 1 -- 172.21.15.129:0/797774237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 msgr2=0x7f46d815e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:15.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.079+0000 7f46df615640 1 --2- 172.21.15.129:0/797774237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d815e2d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e50 tx=0x7f46d002bf60 comp rx=0 tx=0).stop 2024-03-23T16:01:15.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.079+0000 7f46df615640 1 -- 172.21.15.129:0/797774237 shutdown_connections 2024-03-23T16:01:15.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.079+0000 7f46df615640 1 --2- 172.21.15.129:0/797774237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d815e2d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.079+0000 7f46df615640 1 --2- 172.21.15.129:0/797774237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.079+0000 7f46df615640 1 -- 172.21.15.129:0/797774237 >> 172.21.15.129:0/797774237 conn(0x7f46d80cfb20 msgr2=0x7f46d80cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:15.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.080+0000 7f46df615640 1 -- 172.21.15.129:0/797774237 shutdown_connections 2024-03-23T16:01:15.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.080+0000 7f46df615640 1 -- 172.21.15.129:0/797774237 wait complete. 2024-03-23T16:01:15.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.081+0000 7f46df615640 1 Processor -- start 2024-03-23T16:01:15.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.081+0000 7f46df615640 1 -- start start 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46df615640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46d806c030 0x7f46d8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46df615640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d80c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46df615640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d81f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46df615640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46d80c62c0 con 0x7f46d815de40 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46df615640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81f1530 con 0x7f46d806c030 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46df615640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f46d80c3230 con 0x7f46d806c410 2024-03-23T16:01:15.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46ddbb2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d81f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:15.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46dcbb0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d80c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:15.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46ddbb2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d81f1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60902/0 (socket says 172.21.15.129:60902) 2024-03-23T16:01:15.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46ddbb2640 1 -- 172.21.15.129:0/3932533432 learned_addr learned my addr 172.21.15.129:0/3932533432 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:15.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.083+0000 7f46dd3b1640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46d806c030 0x7f46d8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:15.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.087+0000 7f46ddbb2640 1 -- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 msgr2=0x7f46d80c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:15.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.087+0000 7f46ddbb2640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d80c7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.087+0000 7f46ddbb2640 1 -- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46d806c030 msgr2=0x7f46d8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:15.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.087+0000 7f46ddbb2640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46d806c030 0x7f46d8155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.087+0000 7f46ddbb2640 1 -- 172.21.15.129:0/3932533432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d0009b30 con 0x7f46d815de40 2024-03-23T16:01:15.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.090+0000 7f46ddbb2640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d81f1950 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e50 tx=0x7f46d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:15.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.093+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d003e450 con 0x7f46d815de40 2024-03-23T16:01:15.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.093+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d003e5b0 con 0x7f46d815de40 2024-03-23T16:01:15.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.093+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d8157980 con 0x7f46d815de40 2024-03-23T16:01:15.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.093+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d003e960 con 0x7f46d815de40 2024-03-23T16:01:15.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.093+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d815aba0 con 0x7f46d815de40 2024-03-23T16:01:15.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.094+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d80cb700 con 0x7f46d815de40 2024-03-23T16:01:15.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.347+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f46d003eac0 con 0x7f46d815de40 2024-03-23T16:01:15.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.348+0000 7f46cdffb640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f46bc06b860 0x7f46bc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:15.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.348+0000 7f46dd3b1640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f46bc06b860 0x7f46bc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:15.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.348+0000 7f46dd3b1640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f46bc06b860 0x7f46bc06dd20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f46d40016e0 tx=0x7f46d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:15.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.384+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f46d007bb20 con 0x7f46d815de40 2024-03-23T16:01:15.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.384+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f46d0077dc0 con 0x7f46d815de40 2024-03-23T16:01:15.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.611+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46d81f1670 con 0x7f46d815de40 2024-03-23T16:01:15.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.661+0000 7f46cdffb640 1 -- 172.21.15.129:0/3932533432 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f46d0043090 con 0x7f46d815de40 2024-03-23T16:01:15.663 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:15.663 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:15.663 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:15.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f46bc06b860 msgr2=0x7f46bc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:15.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f46bc06b860 0x7f46bc06dd20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f46d40016e0 tx=0x7f46d4009040 comp rx=0 tx=0).stop 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 msgr2=0x7f46d81f1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d81f1950 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e50 tx=0x7f46d0004a40 comp rx=0 tx=0).stop 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 shutdown_connections 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46d815de40 0x7f46d81f1950 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46d806c410 0x7f46d80c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f46bc06b860 0x7f46bc06dd20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 --2- 172.21.15.129:0/3932533432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46d806c030 0x7f46d8155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 >> 172.21.15.129:0/3932533432 conn(0x7f46d80cfb20 msgr2=0x7f46d815d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 shutdown_connections 2024-03-23T16:01:15.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:15.663+0000 7f46df615640 1 -- 172.21.15.129:0/3932533432 wait complete. 2024-03-23T16:01:16.281 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:16.281 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:17.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:17.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.388+0000 7f93bccc0640 1 Processor -- start 2024-03-23T16:01:17.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.388+0000 7f93bccc0640 1 -- start start 2024-03-23T16:01:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.389+0000 7f93bccc0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80ceb00 0x7f93b815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.389+0000 7f93bccc0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b815b290 con 0x7f93b80cef40 2024-03-23T16:01:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.389+0000 7f93bccc0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b80ce800 con 0x7f93b80ccd60 2024-03-23T16:01:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.389+0000 7f93bccc0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93b8154a70 con 0x7f93b80ceb00 2024-03-23T16:01:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.389+0000 7f93b6d9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f93b80cef40 0x7f93b815b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48024/0 (socket says 172.21.15.129:48024) 2024-03-23T16:01:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.389+0000 7f93b6d9d640 1 -- 172.21.15.129:0/3800934826 learned_addr learned my addr 172.21.15.129:0/3800934826 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.390+0000 7f93add9b640 1 --2- 172.21.15.129:0/3800934826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80ceb00 0x7f93b815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.392+0000 7f93add9b640 1 -- 172.21.15.129:0/3800934826 >> v1:172.21.15.157:6789/0 conn(0x7f93b80ccd60 legacy=0x7f93b806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.392+0000 7f93add9b640 1 -- 172.21.15.129:0/3800934826 >> v1:172.21.15.129:6789/0 conn(0x7f93b80cef40 legacy=0x7f93b815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.392+0000 7f93add9b640 1 -- 172.21.15.129:0/3800934826 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b80cd950 con 0x7f93b80ceb00 2024-03-23T16:01:17.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.395+0000 7f93add9b640 1 --2- 172.21.15.129:0/3800934826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80ceb00 0x7f93b815ad50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93a00097d0 tx=0x7f93a002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=737cb9adb5137501 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:17.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.397+0000 7f93b5d9b640 1 -- 172.21.15.129:0/3800934826 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0030880 con 0x7f93b80ceb00 2024-03-23T16:01:17.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.399+0000 7f93b5d9b640 1 -- 172.21.15.129:0/3800934826 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a00309e0 con 0x7f93b80ceb00 2024-03-23T16:01:17.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93b5d9b640 1 -- 172.21.15.129:0/3800934826 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0030d70 con 0x7f93b80ceb00 2024-03-23T16:01:17.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 -- 172.21.15.129:0/3800934826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80ceb00 msgr2=0x7f93b815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3800934826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80ceb00 0x7f93b815ad50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93a00097d0 tx=0x7f93a002b9c0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 -- 172.21.15.129:0/3800934826 shutdown_connections 2024-03-23T16:01:17.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3800934826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80ceb00 0x7f93b815ad50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 -- 172.21.15.129:0/3800934826 >> 172.21.15.129:0/3800934826 conn(0x7f93b80b1840 msgr2=0x7f93b80b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:17.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 -- 172.21.15.129:0/3800934826 shutdown_connections 2024-03-23T16:01:17.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.400+0000 7f93bccc0640 1 -- 172.21.15.129:0/3800934826 wait complete. 2024-03-23T16:01:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 Processor -- start 2024-03-23T16:01:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 -- start start 2024-03-23T16:01:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b80ccd60 0x7f93b81f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 0x7f93b80c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80cef40 0x7f93b8159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81551e0 con 0x7f93b80ceb00 2024-03-23T16:01:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80c7b00 con 0x7f93b80ccd60 2024-03-23T16:01:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.402+0000 7f93bccc0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93b80c6fc0 con 0x7f93b80cef40 2024-03-23T16:01:17.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.403+0000 7f93add9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 0x7f93b80c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:17.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.403+0000 7f93b6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80cef40 0x7f93b8159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:17.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.403+0000 7f93b659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b80ccd60 0x7f93b81f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:17.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.403+0000 7f93add9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 0x7f93b80c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60910/0 (socket says 172.21.15.129:60910) 2024-03-23T16:01:17.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.403+0000 7f93add9b640 1 -- 172.21.15.129:0/3382794823 learned_addr learned my addr 172.21.15.129:0/3382794823 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:17.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.406+0000 7f93add9b640 1 -- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80cef40 msgr2=0x7f93b8159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.406+0000 7f93add9b640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80cef40 0x7f93b8159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.406+0000 7f93add9b640 1 -- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b80ccd60 msgr2=0x7f93b81f3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.406+0000 7f93add9b640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b80ccd60 0x7f93b81f3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.406+0000 7f93add9b640 1 -- 172.21.15.129:0/3382794823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a0009400 con 0x7f93b80ceb00 2024-03-23T16:01:17.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.409+0000 7f93add9b640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 0x7f93b80c8510 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f93a0000c00 tx=0x7f93a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:17.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.412+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a003e040 con 0x7f93b80ceb00 2024-03-23T16:01:17.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.412+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a0030820 con 0x7f93b80ceb00 2024-03-23T16:01:17.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.412+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0044630 con 0x7f93b80ceb00 2024-03-23T16:01:17.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.412+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b80c7260 con 0x7f93b80ceb00 2024-03-23T16:01:17.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.412+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b80c7d10 con 0x7f93b80ceb00 2024-03-23T16:01:17.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.413+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b804f530 con 0x7f93b80ceb00 2024-03-23T16:01:17.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.665+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f93a0044790 con 0x7f93b80ceb00 2024-03-23T16:01:17.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.666+0000 7f93af7fe640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f939806b810 0x7f939806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:17.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.666+0000 7f93b659c640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f939806b810 0x7f939806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:17.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.666+0000 7f93b659c640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f939806b810 0x7f939806dcd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93a8001780 tx=0x7f93a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:17.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.682+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f93a007bb50 con 0x7f93b80ceb00 2024-03-23T16:01:17.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.682+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f93a0077df0 con 0x7f93b80ceb00 2024-03-23T16:01:17.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.904+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93b80686f0 con 0x7f93b80ceb00 2024-03-23T16:01:17.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.941+0000 7f93af7fe640 1 -- 172.21.15.129:0/3382794823 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f93a0034090 con 0x7f93b80ceb00 2024-03-23T16:01:17.943 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:17.943 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:17.943 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f939806b810 msgr2=0x7f939806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f939806b810 0x7f939806dcd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93a8001780 tx=0x7f93a8009040 comp rx=0 tx=0).stop 2024-03-23T16:01:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 msgr2=0x7f93b80c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 0x7f93b80c8510 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f93a0000c00 tx=0x7f93a0004730 comp rx=0 tx=0).stop 2024-03-23T16:01:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 shutdown_connections 2024-03-23T16:01:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93b80cef40 0x7f93b8159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93b80ceb00 0x7f93b80c8510 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f939806b810 0x7f939806dcd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 --2- 172.21.15.129:0/3382794823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93b80ccd60 0x7f93b81f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 >> 172.21.15.129:0/3382794823 conn(0x7f93b80b1840 msgr2=0x7f93b8154150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 shutdown_connections 2024-03-23T16:01:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:17.943+0000 7f93bccc0640 1 -- 172.21.15.129:0/3382794823 wait complete. 2024-03-23T16:01:18.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:18.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:19.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:19.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.662+0000 7fdec5b4f640 1 Processor -- start 2024-03-23T16:01:19.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.662+0000 7fdec5b4f640 1 -- start start 2024-03-23T16:01:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.663+0000 7fdec5b4f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdec006c030 0x7fdec006c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.663+0000 7fdec5b4f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdec00cc080 0x7fdec015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.663+0000 7fdec5b4f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec006cae0 con 0x7fdec00cbc60 2024-03-23T16:01:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.663+0000 7fdec5b4f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fdec00cabc0 con 0x7fdec006c030 2024-03-23T16:01:19.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.663+0000 7fdec5b4f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdec00caea0 con 0x7fdec00cc080 2024-03-23T16:01:19.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.664+0000 7fdebffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdec00cc080 0x7fdec015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:19.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.664+0000 7fdebf7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdec006c030 0x7fdec006c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:19.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.664+0000 7fdebeffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fdec00cbc60 0x7fdec015d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:48036/0 (socket says 172.21.15.129:48036) 2024-03-23T16:01:19.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.664+0000 7fdebeffd640 1 -- 172.21.15.129:0/2793285007 learned_addr learned my addr 172.21.15.129:0/2793285007 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.668+0000 7fdebffff640 1 -- 172.21.15.129:0/2793285007 >> v2:172.21.15.129:3300/0 conn(0x7fdec006c030 msgr2=0x7fdec006c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.668+0000 7fdebffff640 1 --2- 172.21.15.129:0/2793285007 >> v2:172.21.15.129:3300/0 conn(0x7fdec006c030 0x7fdec006c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.668+0000 7fdebffff640 1 -- 172.21.15.129:0/2793285007 >> v1:172.21.15.129:6789/0 conn(0x7fdec00cbc60 legacy=0x7fdec015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:19.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.668+0000 7fdebffff640 1 -- 172.21.15.129:0/2793285007 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00cb180 con 0x7fdec00cc080 2024-03-23T16:01:19.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.670+0000 7fdebffff640 1 --2- 172.21.15.129:0/2793285007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cc080 0x7fdec015e010 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdec00cf720 tx=0x7fdeb802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b621ab1ae04d0ed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:19.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.675+0000 7fdebe7fc640 1 -- 172.21.15.129:0/2793285007 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb802cc80 con 0x7fdec00cc080 2024-03-23T16:01:19.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.675+0000 7fdebe7fc640 1 -- 172.21.15.129:0/2793285007 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb802cde0 con 0x7fdec00cc080 2024-03-23T16:01:19.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.675+0000 7fdebe7fc640 1 -- 172.21.15.129:0/2793285007 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8034660 con 0x7fdec00cc080 2024-03-23T16:01:19.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.676+0000 7fdec5b4f640 1 -- 172.21.15.129:0/2793285007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cc080 msgr2=0x7fdec015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.676+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/2793285007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cc080 0x7fdec015e010 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdec00cf720 tx=0x7fdeb802bed0 comp rx=0 tx=0).stop 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.676+0000 7fdec5b4f640 1 -- 172.21.15.129:0/2793285007 shutdown_connections 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.676+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/2793285007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cc080 0x7fdec015e010 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.676+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/2793285007 >> v2:172.21.15.129:3300/0 conn(0x7fdec006c030 0x7fdec006c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.676+0000 7fdec5b4f640 1 -- 172.21.15.129:0/2793285007 >> 172.21.15.129:0/2793285007 conn(0x7fdec00cfb20 msgr2=0x7fdec00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.677+0000 7fdec5b4f640 1 -- 172.21.15.129:0/2793285007 shutdown_connections 2024-03-23T16:01:19.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.677+0000 7fdec5b4f640 1 -- 172.21.15.129:0/2793285007 wait complete. 2024-03-23T16:01:19.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.678+0000 7fdec5b4f640 1 Processor -- start 2024-03-23T16:01:19.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.679+0000 7fdec5b4f640 1 -- start start 2024-03-23T16:01:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdec5b4f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 0x7fdec015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdec5b4f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cbc60 0x7fdec015c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdec5b4f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdec00cc080 0x7fdec00b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdec5b4f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00b2950 con 0x7fdec006c030 2024-03-23T16:01:19.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdec5b4f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00b34d0 con 0x7fdec00cbc60 2024-03-23T16:01:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdec5b4f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00b3ea0 con 0x7fdec00cc080 2024-03-23T16:01:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdebf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 0x7fdec015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.680+0000 7fdebeffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cbc60 0x7fdec015c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.681+0000 7fdebffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdec00cc080 0x7fdec00b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.681+0000 7fdebf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 0x7fdec015bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60920/0 (socket says 172.21.15.129:60920) 2024-03-23T16:01:19.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.681+0000 7fdebf7fe640 1 -- 172.21.15.129:0/3070847244 learned_addr learned my addr 172.21.15.129:0/3070847244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:19.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.684+0000 7fdebf7fe640 1 -- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdec00cc080 msgr2=0x7fdec00b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:19.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.684+0000 7fdebf7fe640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdec00cc080 0x7fdec00b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:19.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.684+0000 7fdebf7fe640 1 -- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cbc60 msgr2=0x7fdec015c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:19.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.684+0000 7fdebf7fe640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cbc60 0x7fdec015c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:19.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.684+0000 7fdebf7fe640 1 -- 172.21.15.129:0/3070847244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb8009a70 con 0x7fdec006c030 2024-03-23T16:01:19.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.687+0000 7fdebf7fe640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 0x7fdec015bf00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000ea10 tx=0x7fdeb000eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:19.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.691+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0009670 con 0x7fdec006c030 2024-03-23T16:01:19.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.691+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb000ca00 con 0x7fdec006c030 2024-03-23T16:01:19.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.691+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000cdb0 con 0x7fdec006c030 2024-03-23T16:01:19.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.691+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec015c520 con 0x7fdec006c030 2024-03-23T16:01:19.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.692+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec00b2e80 con 0x7fdec006c030 2024-03-23T16:01:19.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.693+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec0075740 con 0x7fdec006c030 2024-03-23T16:01:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.945+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdeb0013470 con 0x7fdec006c030 2024-03-23T16:01:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.945+0000 7fdea7fff640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdeac06b890 0x7fdeac06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:19.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.946+0000 7fdebeffd640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdeac06b890 0x7fdeac06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:19.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.946+0000 7fdebeffd640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdeac06b890 0x7fdeac06dd50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdec00c6110 tx=0x7fdeb4007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.960+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdeb0014e40 con 0x7fdec006c030 2024-03-23T16:01:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:19.960+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdeb00555c0 con 0x7fdec006c030 2024-03-23T16:01:20.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.182+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdec00d0a30 con 0x7fdec006c030 2024-03-23T16:01:20.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.203+0000 7fdea7fff640 1 -- 172.21.15.129:0/3070847244 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7fdeb0017090 con 0x7fdec006c030 2024-03-23T16:01:20.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:20.205 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:20.205 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:20.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdeac06b890 msgr2=0x7fdeac06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:20.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdeac06b890 0x7fdeac06dd50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdec00c6110 tx=0x7fdeb4007e80 comp rx=0 tx=0).stop 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 msgr2=0x7fdec015bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 0x7fdec015bf00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000ea10 tx=0x7fdeb000eed0 comp rx=0 tx=0).stop 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 shutdown_connections 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdec00cc080 0x7fdec00b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdeac06b890 0x7fdeac06dd50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdec00cbc60 0x7fdec015c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 --2- 172.21.15.129:0/3070847244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec006c030 0x7fdec015bf00 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 >> 172.21.15.129:0/3070847244 conn(0x7fdec00cfb20 msgr2=0x7fdec0159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.205+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 shutdown_connections 2024-03-23T16:01:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:20.206+0000 7fdec5b4f640 1 -- 172.21.15.129:0/3070847244 wait complete. 2024-03-23T16:01:20.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:20.824 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:21.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:21.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.925+0000 7f0010e15640 1 Processor -- start 2024-03-23T16:01:21.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.925+0000 7f0010e15640 1 -- start start 2024-03-23T16:01:21.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.926+0000 7f0010e15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c7380 0x7f000c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:21.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.926+0000 7f0010e15640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f000c0c8e90 0x7f000c0c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:21.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.926+0000 7f0010e15640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f000c0c3bd0 con 0x7f000c156f60 2024-03-23T16:01:21.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.926+0000 7f0010e15640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f000c0c7ca0 con 0x7f000c0c8e90 2024-03-23T16:01:21.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.926+0000 7f0010e15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f000c0c3740 con 0x7f000c0c7380 2024-03-23T16:01:21.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.927+0000 7f000a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c7380 0x7f000c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:21.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.927+0000 7f000ad9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f000c156f60 0x7f000c157370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42524/0 (socket says 172.21.15.129:42524) 2024-03-23T16:01:21.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.927+0000 7f000ad9d640 1 -- 172.21.15.129:0/4120867578 learned_addr learned my addr 172.21.15.129:0/4120867578 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:21.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.930+0000 7f000a59c640 1 -- 172.21.15.129:0/4120867578 >> v2:172.21.15.129:3300/0 conn(0x7f000c0c8e90 msgr2=0x7f000c0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:21.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.930+0000 7f000a59c640 1 --2- 172.21.15.129:0/4120867578 >> v2:172.21.15.129:3300/0 conn(0x7f000c0c8e90 0x7f000c0c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:21.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.930+0000 7f000a59c640 1 -- 172.21.15.129:0/4120867578 >> v1:172.21.15.157:6789/0 conn(0x7f000c156f60 legacy=0x7f000c157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:21.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.930+0000 7f000a59c640 1 -- 172.21.15.129:0/4120867578 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000c0c80c0 con 0x7f000c0c7380 2024-03-23T16:01:21.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.932+0000 7f000a59c640 1 --2- 172.21.15.129:0/4120867578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c7380 0x7f000c0c7760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efffc0097a0 tx=0x7efffc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b59845cce9bedbb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:21.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.936+0000 7f000959a640 1 -- 172.21.15.129:0/4120867578 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc030610 con 0x7f000c0c7380 2024-03-23T16:01:21.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.936+0000 7f000959a640 1 -- 172.21.15.129:0/4120867578 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efffc030770 con 0x7f000c0c7380 2024-03-23T16:01:21.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.936+0000 7f000959a640 1 -- 172.21.15.129:0/4120867578 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc030b00 con 0x7f000c0c7380 2024-03-23T16:01:21.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 -- 172.21.15.129:0/4120867578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c7380 msgr2=0x7f000c0c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:21.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 --2- 172.21.15.129:0/4120867578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c7380 0x7f000c0c7760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efffc0097a0 tx=0x7efffc02b960 comp rx=0 tx=0).stop 2024-03-23T16:01:21.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 -- 172.21.15.129:0/4120867578 shutdown_connections 2024-03-23T16:01:21.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 --2- 172.21.15.129:0/4120867578 >> v2:172.21.15.129:3300/0 conn(0x7f000c0c8e90 0x7f000c0c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:21.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 --2- 172.21.15.129:0/4120867578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c7380 0x7f000c0c7760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:21.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 -- 172.21.15.129:0/4120867578 >> 172.21.15.129:0/4120867578 conn(0x7f000c0c8370 msgr2=0x7f000c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:21.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 -- 172.21.15.129:0/4120867578 shutdown_connections 2024-03-23T16:01:21.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.937+0000 7f0010e15640 1 -- 172.21.15.129:0/4120867578 wait complete. 2024-03-23T16:01:21.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.939+0000 7f0010e15640 1 Processor -- start 2024-03-23T16:01:21.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.939+0000 7f0010e15640 1 -- start start 2024-03-23T16:01:21.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.940+0000 7f0010e15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000c0c7380 0x7f000c146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.940+0000 7f0010e15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 0x7f000c13ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.940+0000 7f0010e15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c156f60 0x7f000c13d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.940+0000 7f0010e15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f000c0c9fd0 con 0x7f000c156f60 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.941+0000 7f0010e15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f000c0ca270 con 0x7f000c0c7380 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.941+0000 7f0010e15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f000c0ca550 con 0x7f000c0c8e90 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.941+0000 7f0009d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 0x7f000c13ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:21.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.941+0000 7f000a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000c0c7380 0x7f000c146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:21.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.941+0000 7f0009d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 0x7f000c13ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:47310/0 (socket says 172.21.15.129:47310) 2024-03-23T16:01:21.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.941+0000 7f0009d9b640 1 -- 172.21.15.129:0/1569682239 learned_addr learned my addr 172.21.15.129:0/1569682239 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:21.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.943+0000 7f000ad9d640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c156f60 0x7f000c13d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:21.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.945+0000 7f0009d9b640 1 -- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000c0c7380 msgr2=0x7f000c146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:21.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.945+0000 7f0009d9b640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000c0c7380 0x7f000c146d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:21.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.945+0000 7f0009d9b640 1 -- 172.21.15.129:0/1569682239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c156f60 msgr2=0x7f000c13d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:21.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.945+0000 7f0009d9b640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c156f60 0x7f000c13d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:21.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.945+0000 7f0009d9b640 1 -- 172.21.15.129:0/1569682239 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efffc009400 con 0x7f000c0c8e90 2024-03-23T16:01:21.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.948+0000 7f0009d9b640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 0x7f000c13ce40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f000000b4a0 tx=0x7f000000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.953+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0000010610 con 0x7f000c0c8e90 2024-03-23T16:01:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.953+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0000010770 con 0x7f000c0c8e90 2024-03-23T16:01:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.953+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0000010b20 con 0x7f000c0c8e90 2024-03-23T16:01:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.953+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f000c0ca830 con 0x7f000c0c8e90 2024-03-23T16:01:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.953+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f000c0cadf0 con 0x7f000c0c8e90 2024-03-23T16:01:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:21.954+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f000c13cb60 con 0x7f000c0c8e90 2024-03-23T16:01:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.251+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0000010c80 con 0x7f000c0c8e90 2024-03-23T16:01:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.251+0000 7efffaffd640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7effec06b810 0x7effec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.251+0000 7f000a59c640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7effec06b810 0x7effec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.252+0000 7f000a59c640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7effec06b810 0x7effec06dcd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efffc036000 tx=0x7efffc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:22.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.257+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f00000588c0 con 0x7f000c0c8e90 2024-03-23T16:01:22.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.276+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0000054b60 con 0x7f000c0c8e90 2024-03-23T16:01:22.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.497+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f000c0627f0 con 0x7f000c0c8e90 2024-03-23T16:01:22.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.512+0000 7efffaffd640 1 -- 172.21.15.129:0/1569682239 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f000001e030 con 0x7f000c0c8e90 2024-03-23T16:01:22.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:22.514 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:22.514 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7effec06b810 msgr2=0x7effec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7effec06b810 0x7effec06dcd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efffc036000 tx=0x7efffc03b040 comp rx=0 tx=0).stop 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 msgr2=0x7f000c13ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 0x7f000c13ce40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f000000b4a0 tx=0x7f000000b960 comp rx=0 tx=0).stop 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 shutdown_connections 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c156f60 0x7f000c13d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000c0c8e90 0x7f000c13ce40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7effec06b810 0x7effec06dcd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 --2- 172.21.15.129:0/1569682239 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000c0c7380 0x7f000c146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 >> 172.21.15.129:0/1569682239 conn(0x7f000c0c8370 msgr2=0x7f000c13f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 shutdown_connections 2024-03-23T16:01:22.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:22.514+0000 7f0010e15640 1 -- 172.21.15.129:0/1569682239 wait complete. 2024-03-23T16:01:23.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:23.131 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:24.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:24.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.232+0000 7f2522dc9640 1 Processor -- start 2024-03-23T16:01:24.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.232+0000 7f2522dc9640 1 -- start start 2024-03-23T16:01:24.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.233+0000 7f2522dc9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f251c0ccd60 0x7f251c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:24.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.233+0000 7f2522dc9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:24.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.233+0000 7f2522dc9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f251c15b290 con 0x7f251c0ceb00 2024-03-23T16:01:24.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.233+0000 7f2522dc9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f251c0ce800 con 0x7f251c0ccd60 2024-03-23T16:01:24.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.233+0000 7f2522dc9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f251c154a70 con 0x7f251c0cef40 2024-03-23T16:01:24.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.234+0000 7f2521366640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:24.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.234+0000 7f2520b65640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f251c0ccd60 0x7f251c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:24.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.234+0000 7f2521366640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c15b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:47326/0 (socket says 172.21.15.129:47326) 2024-03-23T16:01:24.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.234+0000 7f2521366640 1 -- 172.21.15.129:0/595417597 learned_addr learned my addr 172.21.15.129:0/595417597 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:24.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.238+0000 7f2521366640 1 -- 172.21.15.129:0/595417597 >> v2:172.21.15.157:3300/0 conn(0x7f251c0ccd60 msgr2=0x7f251c15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:24.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.238+0000 7f2521366640 1 --2- 172.21.15.129:0/595417597 >> v2:172.21.15.157:3300/0 conn(0x7f251c0ccd60 0x7f251c15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.238+0000 7f2521366640 1 -- 172.21.15.129:0/595417597 >> v1:172.21.15.157:6789/0 conn(0x7f251c0ceb00 legacy=0x7f251c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:24.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.238+0000 7f2521366640 1 -- 172.21.15.129:0/595417597 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c0cd950 con 0x7f251c0cef40 2024-03-23T16:01:24.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.241+0000 7f2521366640 1 --2- 172.21.15.129:0/595417597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c15b8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2514009e50 tx=0x7f251402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1181891a589d155 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:24.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.246+0000 7f251b7fe640 1 -- 172.21.15.129:0/595417597 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251402cb90 con 0x7f251c0cef40 2024-03-23T16:01:24.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.246+0000 7f251b7fe640 1 -- 172.21.15.129:0/595417597 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251402ccf0 con 0x7f251c0cef40 2024-03-23T16:01:24.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.246+0000 7f251b7fe640 1 -- 172.21.15.129:0/595417597 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514034610 con 0x7f251c0cef40 2024-03-23T16:01:24.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 -- 172.21.15.129:0/595417597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 msgr2=0x7f251c15b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:24.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 --2- 172.21.15.129:0/595417597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c15b8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2514009e50 tx=0x7f251402bff0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 -- 172.21.15.129:0/595417597 shutdown_connections 2024-03-23T16:01:24.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 --2- 172.21.15.129:0/595417597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c15b8f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 --2- 172.21.15.129:0/595417597 >> v2:172.21.15.157:3300/0 conn(0x7f251c0ccd60 0x7f251c15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 -- 172.21.15.129:0/595417597 >> 172.21.15.129:0/595417597 conn(0x7f251c0b1840 msgr2=0x7f251c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:24.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 -- 172.21.15.129:0/595417597 shutdown_connections 2024-03-23T16:01:24.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.247+0000 7f2522dc9640 1 -- 172.21.15.129:0/595417597 wait complete. 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.248+0000 7f2522dc9640 1 Processor -- start 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.248+0000 7f2522dc9640 1 -- start start 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.249+0000 7f2522dc9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251c0ccd60 0x7f251c1e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.249+0000 7f2522dc9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251c0ceb00 0x7f251c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.249+0000 7f2522dc9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c0b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.249+0000 7f2522dc9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f251c155b20 con 0x7f251c0ccd60 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.249+0000 7f2522dc9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1f00d0 con 0x7f251c0ceb00 2024-03-23T16:01:24.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.249+0000 7f2522dc9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f251c0cb660 con 0x7f251c0cef40 2024-03-23T16:01:24.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.250+0000 7f2521366640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c0b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:24.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.250+0000 7f251bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251c0ceb00 0x7f251c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:24.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.250+0000 7f2520b65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251c0ccd60 0x7f251c1e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:24.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.250+0000 7f2521366640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c0b2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:47340/0 (socket says 172.21.15.129:47340) 2024-03-23T16:01:24.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.250+0000 7f2521366640 1 -- 172.21.15.129:0/3625264905 learned_addr learned my addr 172.21.15.129:0/3625264905 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:24.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.254+0000 7f2521366640 1 -- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251c0ceb00 msgr2=0x7f251c152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:24.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.254+0000 7f2521366640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251c0ceb00 0x7f251c152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.254+0000 7f2521366640 1 -- 172.21.15.129:0/3625264905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251c0ccd60 msgr2=0x7f251c1e48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:24.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.254+0000 7f2521366640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251c0ccd60 0x7f251c1e48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.254+0000 7f2521366640 1 -- 172.21.15.129:0/3625264905 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2514009b30 con 0x7f251c0cef40 2024-03-23T16:01:24.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.257+0000 7f2521366640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c0b2ec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2514009e50 tx=0x7f2514004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:24.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.260+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514009510 con 0x7f251c0cef40 2024-03-23T16:01:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.260+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251c1f02e0 con 0x7f251c0cef40 2024-03-23T16:01:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.260+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f251c1f08a0 con 0x7f251c0cef40 2024-03-23T16:01:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.261+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f251c04f530 con 0x7f251c0cef40 2024-03-23T16:01:24.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.263+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251403e450 con 0x7f251c0cef40 2024-03-23T16:01:24.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.263+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251403e7e0 con 0x7f251c0cef40 2024-03-23T16:01:24.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.529+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f251403e940 con 0x7f251c0cef40 2024-03-23T16:01:24.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.529+0000 7f25197fa640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250006b810 0x7f250006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:24.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.529+0000 7f2520b65640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250006b810 0x7f250006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:24.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.530+0000 7f2520b65640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250006b810 0x7f250006dcd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2510009b40 tx=0x7f2510009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:24.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.534+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f251407b8b0 con 0x7f251c0cef40 2024-03-23T16:01:24.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.545+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2514077b50 con 0x7f251c0cef40 2024-03-23T16:01:24.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.777+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f251c06db50 con 0x7f251c0cef40 2024-03-23T16:01:24.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.880+0000 7f25197fa640 1 -- 172.21.15.129:0/3625264905 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7f251402ce60 con 0x7f251c0cef40 2024-03-23T16:01:24.883 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:24.883 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:24.883 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250006b810 msgr2=0x7f250006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250006b810 0x7f250006dcd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2510009b40 tx=0x7f2510009190 comp rx=0 tx=0).stop 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 msgr2=0x7f251c0b2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c0b2ec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2514009e50 tx=0x7f2514004a40 comp rx=0 tx=0).stop 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 shutdown_connections 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f251c0cef40 0x7f251c0b2ec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f250006b810 0x7f250006dcd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f251c0ceb00 0x7f251c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 --2- 172.21.15.129:0/3625264905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f251c0ccd60 0x7f251c1e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:24.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 >> 172.21.15.129:0/3625264905 conn(0x7f251c0b1840 msgr2=0x7f251c0b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:24.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 shutdown_connections 2024-03-23T16:01:24.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:24.883+0000 7f2522dc9640 1 -- 172.21.15.129:0/3625264905 wait complete. 2024-03-23T16:01:25.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:25.500 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:26.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T16:01:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.606+0000 7efe22993640 1 Processor -- start 2024-03-23T16:01:26.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.606+0000 7efe22993640 1 -- start start 2024-03-23T16:01:26.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.607+0000 7efe22993640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efe1c0ceb00 0x7efe1c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:26.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.607+0000 7efe22993640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe1c0cef40 0x7efe1c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:26.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.607+0000 7efe22993640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe1c15b290 con 0x7efe1c0ccd60 2024-03-23T16:01:26.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.607+0000 7efe22993640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7efe1c0ce800 con 0x7efe1c0cef40 2024-03-23T16:01:26.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.607+0000 7efe22993640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efe1c154a70 con 0x7efe1c0ceb00 2024-03-23T16:01:26.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.608+0000 7efe1b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efe1c0ceb00 0x7efe1c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:26.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.608+0000 7efe1b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efe1c0ceb00 0x7efe1c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52106/0 (socket says 172.21.15.129:52106) 2024-03-23T16:01:26.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.608+0000 7efe1b7fe640 1 -- 172.21.15.129:0/2934534025 learned_addr learned my addr 172.21.15.129:0/2934534025 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:26.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.609+0000 7efe20f30640 1 --2- 172.21.15.129:0/2934534025 >> v2:172.21.15.129:3300/0 conn(0x7efe1c0cef40 0x7efe1c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.611+0000 7efe1b7fe640 1 -- 172.21.15.129:0/2934534025 >> v2:172.21.15.129:3300/0 conn(0x7efe1c0cef40 msgr2=0x7efe1c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.611+0000 7efe1b7fe640 1 --2- 172.21.15.129:0/2934534025 >> v2:172.21.15.129:3300/0 conn(0x7efe1c0cef40 0x7efe1c15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.611+0000 7efe1b7fe640 1 -- 172.21.15.129:0/2934534025 >> v1:172.21.15.129:6789/0 conn(0x7efe1c0ccd60 legacy=0x7efe1c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.611+0000 7efe1b7fe640 1 -- 172.21.15.129:0/2934534025 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1c0cd950 con 0x7efe1c0ceb00 2024-03-23T16:01:26.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.614+0000 7efe1b7fe640 1 --2- 172.21.15.129:0/2934534025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ceb00 0x7efe1c15ad50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efe1c08d1a0 tx=0x7efe1002ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaa0780b6fd458a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:26.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.618+0000 7efe1affd640 1 -- 172.21.15.129:0/2934534025 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1002f850 con 0x7efe1c0ceb00 2024-03-23T16:01:26.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.618+0000 7efe1affd640 1 -- 172.21.15.129:0/2934534025 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1002f9b0 con 0x7efe1c0ceb00 2024-03-23T16:01:26.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.618+0000 7efe1affd640 1 -- 172.21.15.129:0/2934534025 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1002fd40 con 0x7efe1c0ceb00 2024-03-23T16:01:26.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 -- 172.21.15.129:0/2934534025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ceb00 msgr2=0x7efe1c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:26.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 --2- 172.21.15.129:0/2934534025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ceb00 0x7efe1c15ad50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efe1c08d1a0 tx=0x7efe1002ba30 comp rx=0 tx=0).stop 2024-03-23T16:01:26.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 -- 172.21.15.129:0/2934534025 shutdown_connections 2024-03-23T16:01:26.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 --2- 172.21.15.129:0/2934534025 >> v2:172.21.15.129:3300/0 conn(0x7efe1c0cef40 0x7efe1c15b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 --2- 172.21.15.129:0/2934534025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ceb00 0x7efe1c15ad50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 -- 172.21.15.129:0/2934534025 >> 172.21.15.129:0/2934534025 conn(0x7efe1c0b1840 msgr2=0x7efe1c0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 -- 172.21.15.129:0/2934534025 shutdown_connections 2024-03-23T16:01:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.619+0000 7efe22993640 1 -- 172.21.15.129:0/2934534025 wait complete. 2024-03-23T16:01:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.620+0000 7efe22993640 1 Processor -- start 2024-03-23T16:01:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.620+0000 7efe22993640 1 -- start start 2024-03-23T16:01:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.621+0000 7efe22993640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ccd60 0x7efe1c15c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.621+0000 7efe22993640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 0x7efe1c1f0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.621+0000 7efe22993640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe1c0cef40 0x7efe1c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.621+0000 7efe22993640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1e5220 con 0x7efe1c0ceb00 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.621+0000 7efe22993640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c1f12e0 con 0x7efe1c0ccd60 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.621+0000 7efe22993640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efe1c0caa90 con 0x7efe1c0cef40 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.622+0000 7efe1b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 0x7efe1c1f0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.622+0000 7efe1bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ccd60 0x7efe1c15c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.622+0000 7efe20f30640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe1c0cef40 0x7efe1c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.622+0000 7efe1b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 0x7efe1c1f0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36932/0 (socket says 172.21.15.129:36932) 2024-03-23T16:01:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.622+0000 7efe1b7fe640 1 -- 172.21.15.129:0/875250168 learned_addr learned my addr 172.21.15.129:0/875250168 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:26.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.625+0000 7efe1b7fe640 1 -- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe1c0cef40 msgr2=0x7efe1c15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:26.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.625+0000 7efe1b7fe640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe1c0cef40 0x7efe1c15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:26.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.625+0000 7efe1b7fe640 1 -- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ccd60 msgr2=0x7efe1c15c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:26.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.625+0000 7efe1b7fe640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ccd60 0x7efe1c15c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:26.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.625+0000 7efe1b7fe640 1 -- 172.21.15.129:0/875250168 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe100094d0 con 0x7efe1c0ceb00 2024-03-23T16:01:26.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.628+0000 7efe1b7fe640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 0x7efe1c1f0da0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efe10009960 tx=0x7efe100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:26.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.630+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1003b040 con 0x7efe1c0ceb00 2024-03-23T16:01:26.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.630+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1c15d480 con 0x7efe1c0ceb00 2024-03-23T16:01:26.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.630+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1c15cec0 con 0x7efe1c0ceb00 2024-03-23T16:01:26.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.631+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe1c04f530 con 0x7efe1c0ceb00 2024-03-23T16:01:26.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.634+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe10037c40 con 0x7efe1c0ceb00 2024-03-23T16:01:26.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.634+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1004b460 con 0x7efe1c0ceb00 2024-03-23T16:01:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.896+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efe1004b5c0 con 0x7efe1c0ceb00 2024-03-23T16:01:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.896+0000 7efe18ff9640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe0806b840 0x7efe0806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.896+0000 7efe1bfff640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe0806b840 0x7efe0806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.897+0000 7efe1bfff640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe0806b840 0x7efe0806dd00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0016e0 tx=0x7efe0c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:26.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.919+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efe1007bfd0 con 0x7efe1c0ceb00 2024-03-23T16:01:26.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:26.919+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efe10078270 con 0x7efe1c0ceb00 2024-03-23T16:01:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.146+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe1c06db50 con 0x7efe1c0ceb00 2024-03-23T16:01:27.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.203+0000 7efe18ff9640 1 -- 172.21.15.129:0/875250168 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6285 (secure 0 0 0) 0x7efe1007b930 con 0x7efe1c0ceb00 2024-03-23T16:01:27.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:27.205 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T15:47:26.821471+0000","modified":"2024-03-23T15:49:04.020431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4253,"mds_1":4234,"mds_2":4246,"mds_3":4243,"mds_4":4236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4234":{"gid":4234,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.129:6833/1657051494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1657051494},{"type":"v1","addr":"172.21.15.129:6833","nonce":1657051494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4236":{"gid":4236,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.157:6835/568239467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":568239467},{"type":"v1","addr":"172.21.15.157:6835","nonce":568239467}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974413+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4243":{"gid":4243,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/3584357802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3584357802},{"type":"v1","addr":"172.21.15.157:6839","nonce":3584357802}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:38.953486+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4246":{"gid":4246,"name":"e","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3462143238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3462143238},{"type":"v1","addr":"172.21.15.129:6835","nonce":3462143238}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:48:43.974939+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6837/1954021228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1954021228},{"type":"v1","addr":"172.21.15.129:6837","nonce":1954021228}]},"join_fscid":-1,"laggy_since":"2024-03-23T15:49:04.020430+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4253,"qdb_cluster":[4236,4243,4246,4253]},"id":1}]} 2024-03-23T16:01:27.205 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-03-23T16:01:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe0806b840 msgr2=0x7efe0806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe0806b840 0x7efe0806dd00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0016e0 tx=0x7efe0c009040 comp rx=0 tx=0).stop 2024-03-23T16:01:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 msgr2=0x7efe1c1f0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 0x7efe1c1f0da0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efe10009960 tx=0x7efe100046c0 comp rx=0 tx=0).stop 2024-03-23T16:01:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 shutdown_connections 2024-03-23T16:01:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe1c0cef40 0x7efe1c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe1c0ceb00 0x7efe1c1f0da0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe0806b840 0x7efe0806dd00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 --2- 172.21.15.129:0/875250168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe1c0ccd60 0x7efe1c15c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 >> 172.21.15.129:0/875250168 conn(0x7efe1c0b1840 msgr2=0x7efe1c1540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 shutdown_connections 2024-03-23T16:01:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:27.205+0000 7efe22993640 1 -- 172.21.15.129:0/875250168 wait complete. 2024-03-23T16:01:27.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T15:47:26.821471+0000', 'modified': '2024-03-23T15:49:04.020431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4253, 'mds_1': 4234, 'mds_2': 4246, 'mds_3': 4243, 'mds_4': 4236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4234': {'gid': 4234, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.129:6833/1657051494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1657051494}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1657051494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4236': {'gid': 4236, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.157:6835/568239467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 568239467}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 568239467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974413+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4243': {'gid': 4243, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/3584357802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 3584357802}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 3584357802}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:38.953486+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4246': {'gid': 4246, 'name': 'e', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.129:6835/3462143238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3462143238}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3462143238}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:48:43.974939+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6837/1954021228', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6836', 'nonce': 1954021228}, {'type': 'v1', 'addr': '172.21.15.129:6837', 'nonce': 1954021228}]}, 'join_fscid': -1, 'laggy_since': '2024-03-23T15:49:04.020430+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4253, 'qdb_cluster': [4236, 4243, 4246, 4253]} 2024-03-23T16:01:27.828 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4234:up:resolve 2024-03-23T16:01:28.829 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-23T15:47:26.821471+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4234": { "addr": "172.21.15.129:6833/1657051494", "addrs": { "addrvec": [ { "addr": "172.21.15.129:6832", "nonce": 1657051494, "type": "v2" }, { "addr": "172.21.15.129:6833", "nonce": 1657051494, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4234, "incarnation": 18, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 21 }, "gid_4236": { "addr": "172.21.15.157:6835/568239467", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 568239467, "type": "v2" }, { "addr": "172.21.15.157:6835", "nonce": 568239467, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4236, "incarnation": 16, "join_fscid": -1, "laggy_since": "2024-03-23T15:48:43.974413+0000", "name": "d", "rank": 4, "state": "up:active", "state_seq": 7 }, "gid_4243": { "addr": "172.21.15.157:6839/3584357802", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3584357802, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3584357802, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4243, "incarnation": 14, "join_fscid": -1, "laggy_since": "2024-03-23T15:48:38.953486+0000", "name": "b", "rank": 3, "state": "up:active", "state_seq": 6 }, "gid_4246": { "addr": "172.21.15.129:6835/3462143238", "addrs": { "addrvec": [ { "addr": "172.21.15.129:6834", "nonce": 3462143238, "type": "v2" }, { "addr": "172.21.15.129:6835", "nonce": 3462143238, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4246, "incarnation": 12, "join_fscid": -1, "laggy_since": "2024-03-23T15:48:43.974939+0000", "name": "e", "rank": 2, "state": "up:active", "state_seq": 6 }, "gid_4253": { "addr": "172.21.15.129:6837/1954021228", "addrs": { "addrvec": [ { "addr": "172.21.15.129:6836", "nonce": 1954021228, "type": "v2" }, { "addr": "172.21.15.129:6837", "nonce": 1954021228, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4253, "incarnation": 4, "join_fscid": -1, "laggy_since": "2024-03-23T15:49:04.020430+0000", "name": "a", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-23T15:49:04.020431+0000", "qdb_cluster": [ 4236, 4243, 4246, 4253 ], "qdb_leader": 4253, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4253, "mds_1": 4234, "mds_2": 4246, "mds_3": 4243, "mds_4": 4236 } } } ], "standbys": [] } 2024-03-23T16:01:28.830 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T16:01:28.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T16:01:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.930+0000 7f9d5b93e640 1 Processor -- start 2024-03-23T16:01:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.930+0000 7f9d5b93e640 1 -- start start 2024-03-23T16:01:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.931+0000 7f9d5b93e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f9d5415a2e0 0x7f9d5406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.931+0000 7f9d5b93e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540cbcb0 con 0x7f9d5406c0b0 2024-03-23T16:01:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.931+0000 7f9d5b93e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5415b280 con 0x7f9d541544b0 2024-03-23T16:01:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.931+0000 7f9d5b93e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f9d540c5be0 con 0x7f9d5415a2e0 2024-03-23T16:01:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.932+0000 7f9d596da640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9d541544b0 0x7f9d540caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42542/0 (socket says 172.21.15.129:42542) 2024-03-23T16:01:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.932+0000 7f9d596da640 1 -- 172.21.15.129:0/3864378302 learned_addr learned my addr 172.21.15.129:0/3864378302 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.932+0000 7f9d58ed9640 1 --2- 172.21.15.129:0/3864378302 >> v2:172.21.15.129:3300/0 conn(0x7f9d5415a2e0 0x7f9d5406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.934+0000 7f9d4bfff640 1 -- 172.21.15.129:0/3864378302 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751841601 0 0) 0x7f9d5415b280 con 0x7f9d541544b0 2024-03-23T16:01:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.934+0000 7f9d4bfff640 1 -- 172.21.15.129:0/3864378302 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d40003520 con 0x7f9d541544b0 2024-03-23T16:01:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.936+0000 7f9d58ed9640 1 -- 172.21.15.129:0/3864378302 >> v1:172.21.15.157:6789/0 conn(0x7f9d541544b0 legacy=0x7f9d540caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.936+0000 7f9d58ed9640 1 -- 172.21.15.129:0/3864378302 >> v1:172.21.15.129:6789/0 conn(0x7f9d5406c0b0 legacy=0x7f9d540c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.936+0000 7f9d58ed9640 1 -- 172.21.15.129:0/3864378302 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5415a9e0 con 0x7f9d5415a2e0 2024-03-23T16:01:28.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.940+0000 7f9d58ed9640 1 --2- 172.21.15.129:0/3864378302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d5415a2e0 0x7f9d5406bb70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9d440097a0 tx=0x7f9d4402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce304fb9d2c69dc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:28.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.945+0000 7f9d4bfff640 1 -- 172.21.15.129:0/3864378302 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4402f810 con 0x7f9d5415a2e0 2024-03-23T16:01:28.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.945+0000 7f9d4bfff640 1 -- 172.21.15.129:0/3864378302 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4402f970 con 0x7f9d5415a2e0 2024-03-23T16:01:28.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.945+0000 7f9d4bfff640 1 -- 172.21.15.129:0/3864378302 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4402fd00 con 0x7f9d5415a2e0 2024-03-23T16:01:28.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 -- 172.21.15.129:0/3864378302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d5415a2e0 msgr2=0x7f9d5406bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:28.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/3864378302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d5415a2e0 0x7f9d5406bb70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9d440097a0 tx=0x7f9d4402b9f0 comp rx=0 tx=0).stop 2024-03-23T16:01:28.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 -- 172.21.15.129:0/3864378302 shutdown_connections 2024-03-23T16:01:28.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/3864378302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d5415a2e0 0x7f9d5406bb70 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:28.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 -- 172.21.15.129:0/3864378302 >> 172.21.15.129:0/3864378302 conn(0x7f9d54061b50 msgr2=0x7f9d54061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:28.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 -- 172.21.15.129:0/3864378302 shutdown_connections 2024-03-23T16:01:28.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.946+0000 7f9d5b93e640 1 -- 172.21.15.129:0/3864378302 wait complete. 2024-03-23T16:01:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.947+0000 7f9d5b93e640 1 Processor -- start 2024-03-23T16:01:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.947+0000 7f9d5b93e640 1 -- start start 2024-03-23T16:01:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d5b93e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d5406c0b0 0x7f9d54156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d5b93e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 0x7f9d540c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:28.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d5b93e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d5415a2e0 0x7f9d540c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:28.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d5b93e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54155020 con 0x7f9d541544b0 2024-03-23T16:01:28.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d5b93e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54151990 con 0x7f9d5415a2e0 2024-03-23T16:01:28.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d5b93e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9d54151be0 con 0x7f9d5406c0b0 2024-03-23T16:01:28.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d58ed9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 0x7f9d540c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:28.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d59edb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d5415a2e0 0x7f9d540c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:28.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d58ed9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 0x7f9d540c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36954/0 (socket says 172.21.15.129:36954) 2024-03-23T16:01:28.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d58ed9640 1 -- 172.21.15.129:0/1501927418 learned_addr learned my addr 172.21.15.129:0/1501927418 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:28.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.948+0000 7f9d596da640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d5406c0b0 0x7f9d54156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:28.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.952+0000 7f9d58ed9640 1 -- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d5406c0b0 msgr2=0x7f9d54156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:28.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.952+0000 7f9d58ed9640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d5406c0b0 0x7f9d54156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:28.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.952+0000 7f9d58ed9640 1 -- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d5415a2e0 msgr2=0x7f9d540c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:28.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.952+0000 7f9d58ed9640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d5415a2e0 0x7f9d540c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:28.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.952+0000 7f9d58ed9640 1 -- 172.21.15.129:0/1501927418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d44009400 con 0x7f9d541544b0 2024-03-23T16:01:28.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.954+0000 7f9d58ed9640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 0x7f9d540c8300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9d4402c000 tx=0x7f9d44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:28.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.958+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4403c040 con 0x7f9d541544b0 2024-03-23T16:01:28.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.958+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d440378f0 con 0x7f9d541544b0 2024-03-23T16:01:28.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.958+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44037ca0 con 0x7f9d541544b0 2024-03-23T16:01:28.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.958+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d54151ec0 con 0x7f9d541544b0 2024-03-23T16:01:28.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.958+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d540c73f0 con 0x7f9d541544b0 2024-03-23T16:01:28.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:28.960+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d54152100 con 0x7f9d541544b0 2024-03-23T16:01:29.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.213+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9d4404b3b0 con 0x7f9d541544b0 2024-03-23T16:01:29.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.213+0000 7f9d49ffb640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d3806b860 0x7f9d3806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:29.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.214+0000 7f9d596da640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d3806b860 0x7f9d3806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:29.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.214+0000 7f9d596da640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d3806b860 0x7f9d3806dd20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9d500011e0 tx=0x7f9d50007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:29.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.231+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f9d4407ab90 con 0x7f9d541544b0 2024-03-23T16:01:29.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.231+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9d44076e30 con 0x7f9d541544b0 2024-03-23T16:01:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.426+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d540c5800 con 0x7f9d3806b860 2024-03-23T16:01:29.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.433+0000 7f9d49ffb640 1 -- 172.21.15.129:0/1501927418 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251995 (secure 0 0 0) 0x7f9d500014d0 con 0x7f9d3806b860 2024-03-23T16:01:29.438 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:29.443 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T16:01:29.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d3806b860 msgr2=0x7f9d3806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d3806b860 0x7f9d3806dd20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9d500011e0 tx=0x7f9d50007d30 comp rx=0 tx=0).stop 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 msgr2=0x7f9d540c8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 0x7f9d540c8300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9d4402c000 tx=0x7f9d44004730 comp rx=0 tx=0).stop 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 shutdown_connections 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f9d3806b860 0x7f9d3806dd20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9d5415a2e0 0x7f9d540c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d541544b0 0x7f9d540c8300 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 --2- 172.21.15.129:0/1501927418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9d5406c0b0 0x7f9d54156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 >> 172.21.15.129:0/1501927418 conn(0x7f9d54061b50 msgr2=0x7f9d54152910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 shutdown_connections 2024-03-23T16:01:29.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.436+0000 7f9d5b93e640 1 -- 172.21.15.129:0/1501927418 wait complete. 2024-03-23T16:01:29.450 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-03-23T16:01:28.583414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235168,"kb_used_data":2368,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739552,"statfs":{"total":773094113280,"available":772853301248,"internally_reserved":0,"allocated":2424832,"data_stored":1948916,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.366717+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:57:51.366717+0000","last_peered":"2024-03-23T15:57:51.366717+0000","last_clean":"2024-03-23T15:57:51.366717+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:57:51.366717+0000","last_undegraded":"2024-03-23T15:57:51.366717+0000","last_fullsized":"2024-03-23T15:57:51.366717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298822+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:57:51.298822+0000","last_peered":"2024-03-23T15:57:51.298822+0000","last_clean":"2024-03-23T15:57:51.298822+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:57:51.298822+0000","last_undegraded":"2024-03-23T15:57:51.298822+0000","last_fullsized":"2024-03-23T15:57:51.298822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298806+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:57:51.298806+0000","last_peered":"2024-03-23T15:57:51.298806+0000","last_clean":"2024-03-23T15:57:51.298806+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:57:51.298806+0000","last_undegraded":"2024-03-23T15:57:51.298806+0000","last_fullsized":"2024-03-23T15:57:51.298806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.429578+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:59:26.429578+0000","last_peered":"2024-03-23T15:59:26.429578+0000","last_clean":"2024-03-23T15:59:26.429578+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:59:26.429578+0000","last_undegraded":"2024-03-23T15:59:26.429578+0000","last_fullsized":"2024-03-23T15:59:26.429578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364479+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:57:26.364479+0000","last_peered":"2024-03-23T15:57:26.364479+0000","last_clean":"2024-03-23T15:57:26.364479+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:57:26.364479+0000","last_undegraded":"2024-03-23T15:57:26.364479+0000","last_fullsized":"2024-03-23T15:57:26.364479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.420429+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:57:26.420429+0000","last_peered":"2024-03-23T15:57:26.420429+0000","last_clean":"2024-03-23T15:57:26.420429+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:57:26.420429+0000","last_undegraded":"2024-03-23T15:57:26.420429+0000","last_fullsized":"2024-03-23T15:57:26.420429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364457+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:57:26.364457+0000","last_peered":"2024-03-23T15:57:26.364457+0000","last_clean":"2024-03-23T15:57:26.364457+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:57:26.364457+0000","last_undegraded":"2024-03-23T15:57:26.364457+0000","last_fullsized":"2024-03-23T15:57:26.364457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296437+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:57:26.296437+0000","last_peered":"2024-03-23T15:57:26.296437+0000","last_clean":"2024-03-23T15:57:26.296437+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:57:26.296437+0000","last_undegraded":"2024-03-23T15:57:26.296437+0000","last_fullsized":"2024-03-23T15:57:26.296437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.438228+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:57:06.438228+0000","last_peered":"2024-03-23T15:57:06.438228+0000","last_clean":"2024-03-23T15:57:06.438228+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:57:06.438228+0000","last_undegraded":"2024-03-23T15:57:06.438228+0000","last_fullsized":"2024-03-23T15:57:06.438228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362867+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:57:06.362867+0000","last_peered":"2024-03-23T15:57:06.362867+0000","last_clean":"2024-03-23T15:57:06.362867+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:57:06.362867+0000","last_undegraded":"2024-03-23T15:57:06.362867+0000","last_fullsized":"2024-03-23T15:57:06.362867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:46.377367+0000","last_change":"2024-03-23T15:48:04.931538+0000","last_active":"2024-03-23T15:57:46.377367+0000","last_peered":"2024-03-23T15:57:46.377367+0000","last_clean":"2024-03-23T15:57:46.377367+0000","last_became_active":"2024-03-23T15:48:04.931241+0000","last_became_peered":"2024-03-23T15:48:04.931241+0000","last_unstale":"2024-03-23T15:57:46.377367+0000","last_undegraded":"2024-03-23T15:57:46.377367+0000","last_fullsized":"2024-03-23T15:57:46.377367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362823+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:57:06.362823+0000","last_peered":"2024-03-23T15:57:06.362823+0000","last_clean":"2024-03-23T15:57:06.362823+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:57:06.362823+0000","last_undegraded":"2024-03-23T15:57:06.362823+0000","last_fullsized":"2024-03-23T15:57:06.362823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356105+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:57:06.356105+0000","last_peered":"2024-03-23T15:57:06.356105+0000","last_clean":"2024-03-23T15:57:06.356105+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:57:06.356105+0000","last_undegraded":"2024-03-23T15:57:06.356105+0000","last_fullsized":"2024-03-23T15:57:06.356105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366410+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:59:06.366410+0000","last_peered":"2024-03-23T15:59:06.366410+0000","last_clean":"2024-03-23T15:59:06.366410+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:59:06.366410+0000","last_undegraded":"2024-03-23T15:59:06.366410+0000","last_fullsized":"2024-03-23T15:59:06.366410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.305751+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:59:06.305751+0000","last_peered":"2024-03-23T15:59:06.305751+0000","last_clean":"2024-03-23T15:59:06.305751+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:59:06.305751+0000","last_undegraded":"2024-03-23T15:59:06.305751+0000","last_fullsized":"2024-03-23T15:59:06.305751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419506+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:59:06.419506+0000","last_peered":"2024-03-23T15:59:06.419506+0000","last_clean":"2024-03-23T15:59:06.419506+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:59:06.419506+0000","last_undegraded":"2024-03-23T15:59:06.419506+0000","last_fullsized":"2024-03-23T15:59:06.419506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419582+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:59:06.419582+0000","last_peered":"2024-03-23T15:59:06.419582+0000","last_clean":"2024-03-23T15:59:06.419582+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:59:06.419582+0000","last_undegraded":"2024-03-23T15:59:06.419582+0000","last_fullsized":"2024-03-23T15:59:06.419582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383777+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:59:06.383777+0000","last_peered":"2024-03-23T15:59:06.383777+0000","last_clean":"2024-03-23T15:59:06.383777+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:59:06.383777+0000","last_undegraded":"2024-03-23T15:59:06.383777+0000","last_fullsized":"2024-03-23T15:59:06.383777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417473+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:58:41.417473+0000","last_peered":"2024-03-23T15:58:41.417473+0000","last_clean":"2024-03-23T15:58:41.417473+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:58:41.417473+0000","last_undegraded":"2024-03-23T15:58:41.417473+0000","last_fullsized":"2024-03-23T15:58:41.417473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370815+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:58:41.370815+0000","last_peered":"2024-03-23T15:58:41.370815+0000","last_clean":"2024-03-23T15:58:41.370815+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:58:41.370815+0000","last_undegraded":"2024-03-23T15:58:41.370815+0000","last_fullsized":"2024-03-23T15:58:41.370815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446916+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:58:41.446916+0000","last_peered":"2024-03-23T15:58:41.446916+0000","last_clean":"2024-03-23T15:58:41.446916+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:58:41.446916+0000","last_undegraded":"2024-03-23T15:58:41.446916+0000","last_fullsized":"2024-03-23T15:58:41.446916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446527+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:58:41.446527+0000","last_peered":"2024-03-23T15:58:41.446527+0000","last_clean":"2024-03-23T15:58:41.446527+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:58:41.446527+0000","last_undegraded":"2024-03-23T15:58:41.446527+0000","last_fullsized":"2024-03-23T15:58:41.446527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303312+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:58:41.303312+0000","last_peered":"2024-03-23T15:58:41.303312+0000","last_clean":"2024-03-23T15:58:41.303312+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:58:41.303312+0000","last_undegraded":"2024-03-23T15:58:41.303312+0000","last_fullsized":"2024-03-23T15:58:41.303312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415466+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:58:16.415466+0000","last_peered":"2024-03-23T15:58:16.415466+0000","last_clean":"2024-03-23T15:58:16.415466+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:58:16.415466+0000","last_undegraded":"2024-03-23T15:58:16.415466+0000","last_fullsized":"2024-03-23T15:58:16.415466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.301061+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:58:16.301061+0000","last_peered":"2024-03-23T15:58:16.301061+0000","last_clean":"2024-03-23T15:58:16.301061+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:58:16.301061+0000","last_undegraded":"2024-03-23T15:58:16.301061+0000","last_fullsized":"2024-03-23T15:58:16.301061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.424258+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:58:16.424258+0000","last_peered":"2024-03-23T15:58:16.424258+0000","last_clean":"2024-03-23T15:58:16.424258+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:58:16.424258+0000","last_undegraded":"2024-03-23T15:58:16.424258+0000","last_fullsized":"2024-03-23T15:58:16.424258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362212+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:58:16.362212+0000","last_peered":"2024-03-23T15:58:16.362212+0000","last_clean":"2024-03-23T15:58:16.362212+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:58:16.362212+0000","last_undegraded":"2024-03-23T15:58:16.362212+0000","last_fullsized":"2024-03-23T15:58:16.362212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362193+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:58:16.362193+0000","last_peered":"2024-03-23T15:58:16.362193+0000","last_clean":"2024-03-23T15:58:16.362193+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:58:16.362193+0000","last_undegraded":"2024-03-23T15:58:16.362193+0000","last_fullsized":"2024-03-23T15:58:16.362193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298768+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:57:51.298768+0000","last_peered":"2024-03-23T15:57:51.298768+0000","last_clean":"2024-03-23T15:57:51.298768+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:57:51.298768+0000","last_undegraded":"2024-03-23T15:57:51.298768+0000","last_fullsized":"2024-03-23T15:57:51.298768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413315+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:57:51.413315+0000","last_peered":"2024-03-23T15:57:51.413315+0000","last_clean":"2024-03-23T15:57:51.413315+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:57:51.413315+0000","last_undegraded":"2024-03-23T15:57:51.413315+0000","last_fullsized":"2024-03-23T15:57:51.413315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377740+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:57:51.377740+0000","last_peered":"2024-03-23T15:57:51.377740+0000","last_clean":"2024-03-23T15:57:51.377740+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:57:51.377740+0000","last_undegraded":"2024-03-23T15:57:51.377740+0000","last_fullsized":"2024-03-23T15:57:51.377740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.422269+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:57:51.422269+0000","last_peered":"2024-03-23T15:57:51.422269+0000","last_clean":"2024-03-23T15:57:51.422269+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:57:51.422269+0000","last_undegraded":"2024-03-23T15:57:51.422269+0000","last_fullsized":"2024-03-23T15:57:51.422269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440099+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:57:26.440099+0000","last_peered":"2024-03-23T15:57:26.440099+0000","last_clean":"2024-03-23T15:57:26.440099+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:57:26.440099+0000","last_undegraded":"2024-03-23T15:57:26.440099+0000","last_fullsized":"2024-03-23T15:57:26.440099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296495+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:57:26.296495+0000","last_peered":"2024-03-23T15:57:26.296495+0000","last_clean":"2024-03-23T15:57:26.296495+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:57:26.296495+0000","last_undegraded":"2024-03-23T15:57:26.296495+0000","last_fullsized":"2024-03-23T15:57:26.296495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440553+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:57:26.440553+0000","last_peered":"2024-03-23T15:57:26.440553+0000","last_clean":"2024-03-23T15:57:26.440553+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:57:26.440553+0000","last_undegraded":"2024-03-23T15:57:26.440553+0000","last_fullsized":"2024-03-23T15:57:26.440553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.411363+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:57:26.411363+0000","last_peered":"2024-03-23T15:57:26.411363+0000","last_clean":"2024-03-23T15:57:26.411363+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:57:26.411363+0000","last_undegraded":"2024-03-23T15:57:26.411363+0000","last_fullsized":"2024-03-23T15:57:26.411363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356066+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:57:06.356066+0000","last_peered":"2024-03-23T15:57:06.356066+0000","last_clean":"2024-03-23T15:57:06.356066+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:57:06.356066+0000","last_undegraded":"2024-03-23T15:57:06.356066+0000","last_fullsized":"2024-03-23T15:57:06.356066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294507+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:57:06.294507+0000","last_peered":"2024-03-23T15:57:06.294507+0000","last_clean":"2024-03-23T15:57:06.294507+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:57:06.294507+0000","last_undegraded":"2024-03-23T15:57:06.294507+0000","last_fullsized":"2024-03-23T15:57:06.294507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.409374+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:57:06.409374+0000","last_peered":"2024-03-23T15:57:06.409374+0000","last_clean":"2024-03-23T15:57:06.409374+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:57:06.409374+0000","last_undegraded":"2024-03-23T15:57:06.409374+0000","last_fullsized":"2024-03-23T15:57:06.409374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.439044+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:57:06.439044+0000","last_peered":"2024-03-23T15:57:06.439044+0000","last_clean":"2024-03-23T15:57:06.439044+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:57:06.439044+0000","last_undegraded":"2024-03-23T15:57:06.439044+0000","last_fullsized":"2024-03-23T15:57:06.439044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.438259+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:57:06.438259+0000","last_peered":"2024-03-23T15:57:06.438259+0000","last_clean":"2024-03-23T15:57:06.438259+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:57:06.438259+0000","last_undegraded":"2024-03-23T15:57:06.438259+0000","last_fullsized":"2024-03-23T15:57:06.438259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.418592+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:57:06.418592+0000","last_peered":"2024-03-23T15:57:06.418592+0000","last_clean":"2024-03-23T15:57:06.418592+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:57:06.418592+0000","last_undegraded":"2024-03-23T15:57:06.418592+0000","last_fullsized":"2024-03-23T15:57:06.418592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362879+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:57:06.362879+0000","last_peered":"2024-03-23T15:57:06.362879+0000","last_clean":"2024-03-23T15:57:06.362879+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:57:06.362879+0000","last_undegraded":"2024-03-23T15:57:06.362879+0000","last_fullsized":"2024-03-23T15:57:06.362879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294544+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:57:06.294544+0000","last_peered":"2024-03-23T15:57:06.294544+0000","last_clean":"2024-03-23T15:57:06.294544+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:57:06.294544+0000","last_undegraded":"2024-03-23T15:57:06.294544+0000","last_fullsized":"2024-03-23T15:57:06.294544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.374127+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:57:06.374127+0000","last_peered":"2024-03-23T15:57:06.374127+0000","last_clean":"2024-03-23T15:57:06.374127+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:57:06.374127+0000","last_undegraded":"2024-03-23T15:57:06.374127+0000","last_fullsized":"2024-03-23T15:57:06.374127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419569+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:59:06.419569+0000","last_peered":"2024-03-23T15:59:06.419569+0000","last_clean":"2024-03-23T15:59:06.419569+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:59:06.419569+0000","last_undegraded":"2024-03-23T15:59:06.419569+0000","last_fullsized":"2024-03-23T15:59:06.419569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.428095+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:59:06.428095+0000","last_peered":"2024-03-23T15:59:06.428095+0000","last_clean":"2024-03-23T15:59:06.428095+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:59:06.428095+0000","last_undegraded":"2024-03-23T15:59:06.428095+0000","last_fullsized":"2024-03-23T15:59:06.428095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362853+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:57:06.362853+0000","last_peered":"2024-03-23T15:57:06.362853+0000","last_clean":"2024-03-23T15:57:06.362853+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:57:06.362853+0000","last_undegraded":"2024-03-23T15:57:06.362853+0000","last_fullsized":"2024-03-23T15:57:06.362853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383756+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:59:06.383756+0000","last_peered":"2024-03-23T15:59:06.383756+0000","last_clean":"2024-03-23T15:59:06.383756+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:59:06.383756+0000","last_undegraded":"2024-03-23T15:59:06.383756+0000","last_fullsized":"2024-03-23T15:59:06.383756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366429+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:59:06.366429+0000","last_peered":"2024-03-23T15:59:06.366429+0000","last_clean":"2024-03-23T15:59:06.366429+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:59:06.366429+0000","last_undegraded":"2024-03-23T15:59:06.366429+0000","last_fullsized":"2024-03-23T15:59:06.366429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419548+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:59:06.419548+0000","last_peered":"2024-03-23T15:59:06.419548+0000","last_clean":"2024-03-23T15:59:06.419548+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:59:06.419548+0000","last_undegraded":"2024-03-23T15:59:06.419548+0000","last_fullsized":"2024-03-23T15:59:06.419548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419558+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:59:06.419558+0000","last_peered":"2024-03-23T15:59:06.419558+0000","last_clean":"2024-03-23T15:59:06.419558+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:59:06.419558+0000","last_undegraded":"2024-03-23T15:59:06.419558+0000","last_fullsized":"2024-03-23T15:59:06.419558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.373056+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:59:06.373056+0000","last_peered":"2024-03-23T15:59:06.373056+0000","last_clean":"2024-03-23T15:59:06.373056+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:59:06.373056+0000","last_undegraded":"2024-03-23T15:59:06.373056+0000","last_fullsized":"2024-03-23T15:59:06.373056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.448540+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:59:06.448540+0000","last_peered":"2024-03-23T15:59:06.448540+0000","last_clean":"2024-03-23T15:59:06.448540+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:59:06.448540+0000","last_undegraded":"2024-03-23T15:59:06.448540+0000","last_fullsized":"2024-03-23T15:59:06.448540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.449312+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:59:06.449312+0000","last_peered":"2024-03-23T15:59:06.449312+0000","last_clean":"2024-03-23T15:59:06.449312+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:59:06.449312+0000","last_undegraded":"2024-03-23T15:59:06.449312+0000","last_fullsized":"2024-03-23T15:59:06.449312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419535+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:59:06.419535+0000","last_peered":"2024-03-23T15:59:06.419535+0000","last_clean":"2024-03-23T15:59:06.419535+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:59:06.419535+0000","last_undegraded":"2024-03-23T15:59:06.419535+0000","last_fullsized":"2024-03-23T15:59:06.419535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370893+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:58:41.370893+0000","last_peered":"2024-03-23T15:58:41.370893+0000","last_clean":"2024-03-23T15:58:41.370893+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:58:41.370893+0000","last_undegraded":"2024-03-23T15:58:41.370893+0000","last_fullsized":"2024-03-23T15:58:41.370893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.426198+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:58:41.426198+0000","last_peered":"2024-03-23T15:58:41.426198+0000","last_clean":"2024-03-23T15:58:41.426198+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:58:41.426198+0000","last_undegraded":"2024-03-23T15:58:41.426198+0000","last_fullsized":"2024-03-23T15:58:41.426198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446943+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:58:41.446943+0000","last_peered":"2024-03-23T15:58:41.446943+0000","last_clean":"2024-03-23T15:58:41.446943+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:58:41.446943+0000","last_undegraded":"2024-03-23T15:58:41.446943+0000","last_fullsized":"2024-03-23T15:58:41.446943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370879+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:58:41.370879+0000","last_peered":"2024-03-23T15:58:41.370879+0000","last_clean":"2024-03-23T15:58:41.370879+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:58:41.370879+0000","last_undegraded":"2024-03-23T15:58:41.370879+0000","last_fullsized":"2024-03-23T15:58:41.370879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303343+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:58:41.303343+0000","last_peered":"2024-03-23T15:58:41.303343+0000","last_clean":"2024-03-23T15:58:41.303343+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:58:41.303343+0000","last_undegraded":"2024-03-23T15:58:41.303343+0000","last_fullsized":"2024-03-23T15:58:41.303343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370845+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:58:41.370845+0000","last_peered":"2024-03-23T15:58:41.370845+0000","last_clean":"2024-03-23T15:58:41.370845+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:58:41.370845+0000","last_undegraded":"2024-03-23T15:58:41.370845+0000","last_fullsized":"2024-03-23T15:58:41.370845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.364338+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:58:41.364338+0000","last_peered":"2024-03-23T15:58:41.364338+0000","last_clean":"2024-03-23T15:58:41.364338+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:58:41.364338+0000","last_undegraded":"2024-03-23T15:58:41.364338+0000","last_fullsized":"2024-03-23T15:58:41.364338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417465+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:58:41.417465+0000","last_peered":"2024-03-23T15:58:41.417465+0000","last_clean":"2024-03-23T15:58:41.417465+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:58:41.417465+0000","last_undegraded":"2024-03-23T15:58:41.417465+0000","last_fullsized":"2024-03-23T15:58:41.417465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:16.449240+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:59:16.449240+0000","last_peered":"2024-03-23T15:59:16.449240+0000","last_clean":"2024-03-23T15:59:16.449240+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:59:16.449240+0000","last_undegraded":"2024-03-23T15:59:16.449240+0000","last_fullsized":"2024-03-23T15:59:16.449240+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413350+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:57:51.413350+0000","last_peered":"2024-03-23T15:57:51.413350+0000","last_clean":"2024-03-23T15:57:51.413350+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:57:51.413350+0000","last_undegraded":"2024-03-23T15:57:51.413350+0000","last_fullsized":"2024-03-23T15:57:51.413350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442763+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:57:51.442763+0000","last_peered":"2024-03-23T15:57:51.442763+0000","last_clean":"2024-03-23T15:57:51.442763+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:57:51.442763+0000","last_undegraded":"2024-03-23T15:57:51.442763+0000","last_fullsized":"2024-03-23T15:57:51.442763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.421181+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:59:26.421181+0000","last_peered":"2024-03-23T15:59:26.421181+0000","last_clean":"2024-03-23T15:59:26.421181+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:59:26.421181+0000","last_undegraded":"2024-03-23T15:59:26.421181+0000","last_fullsized":"2024-03-23T15:59:26.421181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.359976+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:57:51.359976+0000","last_peered":"2024-03-23T15:57:51.359976+0000","last_clean":"2024-03-23T15:57:51.359976+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:57:51.359976+0000","last_undegraded":"2024-03-23T15:57:51.359976+0000","last_fullsized":"2024-03-23T15:57:51.359976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377759+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:57:51.377759+0000","last_peered":"2024-03-23T15:57:51.377759+0000","last_clean":"2024-03-23T15:57:51.377759+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:57:51.377759+0000","last_undegraded":"2024-03-23T15:57:51.377759+0000","last_fullsized":"2024-03-23T15:57:51.377759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307635+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:59:26.307635+0000","last_peered":"2024-03-23T15:59:26.307635+0000","last_clean":"2024-03-23T15:59:26.307635+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:59:26.307635+0000","last_undegraded":"2024-03-23T15:59:26.307635+0000","last_fullsized":"2024-03-23T15:59:26.307635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413338+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:57:51.413338+0000","last_peered":"2024-03-23T15:57:51.413338+0000","last_clean":"2024-03-23T15:57:51.413338+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:57:51.413338+0000","last_undegraded":"2024-03-23T15:57:51.413338+0000","last_fullsized":"2024-03-23T15:57:51.413338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.366687+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:57:51.366687+0000","last_peered":"2024-03-23T15:57:51.366687+0000","last_clean":"2024-03-23T15:57:51.366687+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:57:51.366687+0000","last_undegraded":"2024-03-23T15:57:51.366687+0000","last_fullsized":"2024-03-23T15:57:51.366687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298792+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:57:51.298792+0000","last_peered":"2024-03-23T15:57:51.298792+0000","last_clean":"2024-03-23T15:57:51.298792+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:57:51.298792+0000","last_undegraded":"2024-03-23T15:57:51.298792+0000","last_fullsized":"2024-03-23T15:57:51.298792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.422291+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:57:51.422291+0000","last_peered":"2024-03-23T15:57:51.422291+0000","last_clean":"2024-03-23T15:57:51.422291+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:57:51.422291+0000","last_undegraded":"2024-03-23T15:57:51.422291+0000","last_fullsized":"2024-03-23T15:57:51.422291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413367+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:57:51.413367+0000","last_peered":"2024-03-23T15:57:51.413367+0000","last_clean":"2024-03-23T15:57:51.413367+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:57:51.413367+0000","last_undegraded":"2024-03-23T15:57:51.413367+0000","last_fullsized":"2024-03-23T15:57:51.413367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.360026+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:57:51.360026+0000","last_peered":"2024-03-23T15:57:51.360026+0000","last_clean":"2024-03-23T15:57:51.360026+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:57:51.360026+0000","last_undegraded":"2024-03-23T15:57:51.360026+0000","last_fullsized":"2024-03-23T15:57:51.360026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444935+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:58:16.444935+0000","last_peered":"2024-03-23T15:58:16.444935+0000","last_clean":"2024-03-23T15:58:16.444935+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:58:16.444935+0000","last_undegraded":"2024-03-23T15:58:16.444935+0000","last_fullsized":"2024-03-23T15:58:16.444935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444947+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:58:16.444947+0000","last_peered":"2024-03-23T15:58:16.444947+0000","last_clean":"2024-03-23T15:58:16.444947+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:58:16.444947+0000","last_undegraded":"2024-03-23T15:58:16.444947+0000","last_fullsized":"2024-03-23T15:58:16.444947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415523+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:58:16.415523+0000","last_peered":"2024-03-23T15:58:16.415523+0000","last_clean":"2024-03-23T15:58:16.415523+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:58:16.415523+0000","last_undegraded":"2024-03-23T15:58:16.415523+0000","last_fullsized":"2024-03-23T15:58:16.415523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444921+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:58:16.444921+0000","last_peered":"2024-03-23T15:58:16.444921+0000","last_clean":"2024-03-23T15:58:16.444921+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:58:16.444921+0000","last_undegraded":"2024-03-23T15:58:16.444921+0000","last_fullsized":"2024-03-23T15:58:16.444921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.368709+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:58:16.368709+0000","last_peered":"2024-03-23T15:58:16.368709+0000","last_clean":"2024-03-23T15:58:16.368709+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:58:16.368709+0000","last_undegraded":"2024-03-23T15:58:16.368709+0000","last_fullsized":"2024-03-23T15:58:16.368709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415489+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:58:16.415489+0000","last_peered":"2024-03-23T15:58:16.415489+0000","last_clean":"2024-03-23T15:58:16.415489+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:58:16.415489+0000","last_undegraded":"2024-03-23T15:58:16.415489+0000","last_fullsized":"2024-03-23T15:58:16.415489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444897+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:58:16.444897+0000","last_peered":"2024-03-23T15:58:16.444897+0000","last_clean":"2024-03-23T15:58:16.444897+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:58:16.444897+0000","last_undegraded":"2024-03-23T15:58:16.444897+0000","last_fullsized":"2024-03-23T15:58:16.444897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444865+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:58:16.444865+0000","last_peered":"2024-03-23T15:58:16.444865+0000","last_clean":"2024-03-23T15:58:16.444865+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:58:16.444865+0000","last_undegraded":"2024-03-23T15:58:16.444865+0000","last_fullsized":"2024-03-23T15:58:16.444865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362160+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:58:16.362160+0000","last_peered":"2024-03-23T15:58:16.362160+0000","last_clean":"2024-03-23T15:58:16.362160+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:58:16.362160+0000","last_undegraded":"2024-03-23T15:58:16.362160+0000","last_fullsized":"2024-03-23T15:58:16.362160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.381785+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:58:41.381785+0000","last_peered":"2024-03-23T15:58:41.381785+0000","last_clean":"2024-03-23T15:58:41.381785+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:58:41.381785+0000","last_undegraded":"2024-03-23T15:58:41.381785+0000","last_fullsized":"2024-03-23T15:58:41.381785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417439+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:58:41.417439+0000","last_peered":"2024-03-23T15:58:41.417439+0000","last_clean":"2024-03-23T15:58:41.417439+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:58:41.417439+0000","last_undegraded":"2024-03-23T15:58:41.417439+0000","last_fullsized":"2024-03-23T15:58:41.417439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415498+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:58:16.415498+0000","last_peered":"2024-03-23T15:58:16.415498+0000","last_clean":"2024-03-23T15:58:16.415498+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:58:16.415498+0000","last_undegraded":"2024-03-23T15:58:16.415498+0000","last_fullsized":"2024-03-23T15:58:16.415498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.411385+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:57:26.411385+0000","last_peered":"2024-03-23T15:57:26.411385+0000","last_clean":"2024-03-23T15:57:26.411385+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:57:26.411385+0000","last_undegraded":"2024-03-23T15:57:26.411385+0000","last_fullsized":"2024-03-23T15:57:26.411385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364494+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:57:26.364494+0000","last_peered":"2024-03-23T15:57:26.364494+0000","last_clean":"2024-03-23T15:57:26.364494+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:57:26.364494+0000","last_undegraded":"2024-03-23T15:57:26.364494+0000","last_fullsized":"2024-03-23T15:57:26.364494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440130+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:57:26.440130+0000","last_peered":"2024-03-23T15:57:26.440130+0000","last_clean":"2024-03-23T15:57:26.440130+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:57:26.440130+0000","last_undegraded":"2024-03-23T15:57:26.440130+0000","last_fullsized":"2024-03-23T15:57:26.440130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364504+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:57:26.364504+0000","last_peered":"2024-03-23T15:57:26.364504+0000","last_clean":"2024-03-23T15:57:26.364504+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:57:26.364504+0000","last_undegraded":"2024-03-23T15:57:26.364504+0000","last_fullsized":"2024-03-23T15:57:26.364504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298841+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:57:51.298841+0000","last_peered":"2024-03-23T15:57:51.298841+0000","last_clean":"2024-03-23T15:57:51.298841+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:57:51.298841+0000","last_undegraded":"2024-03-23T15:57:51.298841+0000","last_fullsized":"2024-03-23T15:57:51.298841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307684+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:59:26.307684+0000","last_peered":"2024-03-23T15:59:26.307684+0000","last_clean":"2024-03-23T15:59:26.307684+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:59:26.307684+0000","last_undegraded":"2024-03-23T15:59:26.307684+0000","last_fullsized":"2024-03-23T15:59:26.307684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307673+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:59:26.307673+0000","last_peered":"2024-03-23T15:59:26.307673+0000","last_clean":"2024-03-23T15:59:26.307673+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:59:26.307673+0000","last_undegraded":"2024-03-23T15:59:26.307673+0000","last_fullsized":"2024-03-23T15:59:26.307673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442806+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:57:51.442806+0000","last_peered":"2024-03-23T15:57:51.442806+0000","last_clean":"2024-03-23T15:57:51.442806+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:57:51.442806+0000","last_undegraded":"2024-03-23T15:57:51.442806+0000","last_fullsized":"2024-03-23T15:57:51.442806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575023,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":290816,"data_stored":95068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575022,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":96394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":685208,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":311296,"data_stored":97464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575023,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":95694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":685796,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":96556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T16:01:29.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T16:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.590+0000 7fc3c0bc7640 1 Processor -- start 2024-03-23T16:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.591+0000 7fc3c0bc7640 1 -- start start 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.592+0000 7fc3c0bc7640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc3bc15dba0 0x7fc3bc06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.592+0000 7fc3c0bc7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc070640 con 0x7fc3bc15df80 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.592+0000 7fc3c0bc7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc0c4c00 con 0x7fc3bc0cca30 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.592+0000 7fc3c0bc7640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fc3bc0ccea0 con 0x7fc3bc15dba0 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.593+0000 7fc3ba59c640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fc3bc15dba0 0x7fc3bc06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.593+0000 7fc3b3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc3bc0cca30 0x7fc3bc06ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:42546/0 (socket says 172.21.15.129:42546) 2024-03-23T16:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.593+0000 7fc3b3fff640 1 -- 172.21.15.129:0/3935396764 learned_addr learned my addr 172.21.15.129:0/3935396764 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:29.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.596+0000 7fc3b9d9b640 1 -- 172.21.15.129:0/3935396764 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744127867 0 0) 0x7fc3a4001160 con 0x7fc3bc0cca30 2024-03-23T16:01:29.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.596+0000 7fc3b9d9b640 1 -- 172.21.15.129:0/3935396764 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3a8003520 con 0x7fc3bc0cca30 2024-03-23T16:01:29.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.597+0000 7fc3ba59c640 1 -- 172.21.15.129:0/3935396764 >> v1:172.21.15.157:6789/0 conn(0x7fc3bc0cca30 legacy=0x7fc3bc06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.597+0000 7fc3ba59c640 1 -- 172.21.15.129:0/3935396764 >> v1:172.21.15.129:6789/0 conn(0x7fc3bc15df80 legacy=0x7fc3bc1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.597+0000 7fc3ba59c640 1 -- 172.21.15.129:0/3935396764 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc06c240 con 0x7fc3bc15dba0 2024-03-23T16:01:29.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.602+0000 7fc3ba59c640 1 --2- 172.21.15.129:0/3935396764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc06cf70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0097d0 tx=0x7fc3ac02b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d90a5acb0d0c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:29.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.607+0000 7fc3b9d9b640 1 -- 172.21.15.129:0/3935396764 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac030890 con 0x7fc3bc15dba0 2024-03-23T16:01:29.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.608+0000 7fc3b9d9b640 1 -- 172.21.15.129:0/3935396764 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac0309f0 con 0x7fc3bc15dba0 2024-03-23T16:01:29.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.608+0000 7fc3b9d9b640 1 -- 172.21.15.129:0/3935396764 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac030d60 con 0x7fc3bc15dba0 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3935396764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc15dba0 msgr2=0x7fc3bc06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3935396764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc06cf70 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0097d0 tx=0x7fc3ac02b9d0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3935396764 shutdown_connections 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3935396764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc06cf70 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3935396764 >> 172.21.15.129:0/3935396764 conn(0x7fc3bc071300 msgr2=0x7fc3bc0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3935396764 shutdown_connections 2024-03-23T16:01:29.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.609+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3935396764 wait complete. 2024-03-23T16:01:29.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.610+0000 7fc3c0bc7640 1 Processor -- start 2024-03-23T16:01:29.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.610+0000 7fc3c0bc7640 1 -- start start 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3c0bc7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc0cca30 0x7fc3bc0c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3c0bc7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc14f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3c0bc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3bc15df80 0x7fc3bc14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3c0bc7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc1e5640 con 0x7fc3bc0cca30 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3c0bc7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc14e8c0 con 0x7fc3bc15dba0 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3c0bc7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc3bc14eba0 con 0x7fc3bc15df80 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3ba59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc0cca30 0x7fc3bc0c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:29.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.611+0000 7fc3b3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc14f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:29.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.612+0000 7fc3bad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3bc15df80 0x7fc3bc14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:29.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.612+0000 7fc3b3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc14f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:52136/0 (socket says 172.21.15.129:52136) 2024-03-23T16:01:29.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.612+0000 7fc3ba59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc0cca30 0x7fc3bc0c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36966/0 (socket says 172.21.15.129:36966) 2024-03-23T16:01:29.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.612+0000 7fc3b3fff640 1 -- 172.21.15.129:0/3207119319 learned_addr learned my addr 172.21.15.129:0/3207119319 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.615+0000 7fc3b3fff640 1 -- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3bc15df80 msgr2=0x7fc3bc14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.615+0000 7fc3b3fff640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3bc15df80 0x7fc3bc14f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.615+0000 7fc3b3fff640 1 -- 172.21.15.129:0/3207119319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc0cca30 msgr2=0x7fc3bc0c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:29.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.615+0000 7fc3b3fff640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc0cca30 0x7fc3bc0c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:29.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.615+0000 7fc3b3fff640 1 -- 172.21.15.129:0/3207119319 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ac009400 con 0x7fc3bc15dba0 2024-03-23T16:01:29.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.618+0000 7fc3b3fff640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc14f030 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4001f00 tx=0x7fc3a400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:29.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.623+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4005d30 con 0x7fc3bc15dba0 2024-03-23T16:01:29.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.623+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a4005e90 con 0x7fc3bc15dba0 2024-03-23T16:01:29.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.623+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3bc1e5900 con 0x7fc3bc15dba0 2024-03-23T16:01:29.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.623+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40072d0 con 0x7fc3bc15dba0 2024-03-23T16:01:29.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.623+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3bc1e5b70 con 0x7fc3bc15dba0 2024-03-23T16:01:29.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.624+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3bc075720 con 0x7fc3bc15dba0 2024-03-23T16:01:29.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.648+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3a4007450 con 0x7fc3bc15dba0 2024-03-23T16:01:29.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.648+0000 7fc3b2ffd640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc39c06b840 0x7fc39c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:29.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.649+0000 7fc3ba59c640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc39c06b840 0x7fc39c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:29.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.649+0000 7fc3ba59c640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc39c06b840 0x7fc39c06dd00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac036000 tx=0x7fc3ac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:29.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.892+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fc3a4058220 con 0x7fc3bc15dba0 2024-03-23T16:01:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:29.904+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3a4057ff0 con 0x7fc3bc15dba0 2024-03-23T16:01:30.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.096+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3bc06c490 con 0x7fc39c06b840 2024-03-23T16:01:30.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.102+0000 7fc3b2ffd640 1 -- 172.21.15.129:0/3207119319 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251995 (secure 0 0 0) 0x7fc3ac040040 con 0x7fc39c06b840 2024-03-23T16:01:30.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:30.108 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T16:01:30.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc39c06b840 msgr2=0x7fc39c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc39c06b840 0x7fc39c06dd00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac036000 tx=0x7fc3ac008030 comp rx=0 tx=0).stop 2024-03-23T16:01:30.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 msgr2=0x7fc3bc14f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc14f030 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4001f00 tx=0x7fc3a400c960 comp rx=0 tx=0).stop 2024-03-23T16:01:30.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 shutdown_connections 2024-03-23T16:01:30.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3bc15df80 0x7fc3bc14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fc39c06b840 0x7fc39c06dd00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3bc15dba0 0x7fc3bc14f030 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 --2- 172.21.15.129:0/3207119319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3bc0cca30 0x7fc3bc0c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 >> 172.21.15.129:0/3207119319 conn(0x7fc3bc071300 msgr2=0x7fc3bc149e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 shutdown_connections 2024-03-23T16:01:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.105+0000 7fc3c0bc7640 1 -- 172.21.15.129:0/3207119319 wait complete. 2024-03-23T16:01:30.118 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-03-23T16:01:28.583414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235168,"kb_used_data":2368,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739552,"statfs":{"total":773094113280,"available":772853301248,"internally_reserved":0,"allocated":2424832,"data_stored":1948916,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.366717+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:57:51.366717+0000","last_peered":"2024-03-23T15:57:51.366717+0000","last_clean":"2024-03-23T15:57:51.366717+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:57:51.366717+0000","last_undegraded":"2024-03-23T15:57:51.366717+0000","last_fullsized":"2024-03-23T15:57:51.366717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298822+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:57:51.298822+0000","last_peered":"2024-03-23T15:57:51.298822+0000","last_clean":"2024-03-23T15:57:51.298822+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:57:51.298822+0000","last_undegraded":"2024-03-23T15:57:51.298822+0000","last_fullsized":"2024-03-23T15:57:51.298822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298806+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:57:51.298806+0000","last_peered":"2024-03-23T15:57:51.298806+0000","last_clean":"2024-03-23T15:57:51.298806+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:57:51.298806+0000","last_undegraded":"2024-03-23T15:57:51.298806+0000","last_fullsized":"2024-03-23T15:57:51.298806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.429578+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:59:26.429578+0000","last_peered":"2024-03-23T15:59:26.429578+0000","last_clean":"2024-03-23T15:59:26.429578+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:59:26.429578+0000","last_undegraded":"2024-03-23T15:59:26.429578+0000","last_fullsized":"2024-03-23T15:59:26.429578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364479+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:57:26.364479+0000","last_peered":"2024-03-23T15:57:26.364479+0000","last_clean":"2024-03-23T15:57:26.364479+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:57:26.364479+0000","last_undegraded":"2024-03-23T15:57:26.364479+0000","last_fullsized":"2024-03-23T15:57:26.364479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.420429+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:57:26.420429+0000","last_peered":"2024-03-23T15:57:26.420429+0000","last_clean":"2024-03-23T15:57:26.420429+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:57:26.420429+0000","last_undegraded":"2024-03-23T15:57:26.420429+0000","last_fullsized":"2024-03-23T15:57:26.420429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364457+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:57:26.364457+0000","last_peered":"2024-03-23T15:57:26.364457+0000","last_clean":"2024-03-23T15:57:26.364457+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:57:26.364457+0000","last_undegraded":"2024-03-23T15:57:26.364457+0000","last_fullsized":"2024-03-23T15:57:26.364457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296437+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:57:26.296437+0000","last_peered":"2024-03-23T15:57:26.296437+0000","last_clean":"2024-03-23T15:57:26.296437+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:57:26.296437+0000","last_undegraded":"2024-03-23T15:57:26.296437+0000","last_fullsized":"2024-03-23T15:57:26.296437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.438228+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:57:06.438228+0000","last_peered":"2024-03-23T15:57:06.438228+0000","last_clean":"2024-03-23T15:57:06.438228+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:57:06.438228+0000","last_undegraded":"2024-03-23T15:57:06.438228+0000","last_fullsized":"2024-03-23T15:57:06.438228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362867+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:57:06.362867+0000","last_peered":"2024-03-23T15:57:06.362867+0000","last_clean":"2024-03-23T15:57:06.362867+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:57:06.362867+0000","last_undegraded":"2024-03-23T15:57:06.362867+0000","last_fullsized":"2024-03-23T15:57:06.362867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:46.377367+0000","last_change":"2024-03-23T15:48:04.931538+0000","last_active":"2024-03-23T15:57:46.377367+0000","last_peered":"2024-03-23T15:57:46.377367+0000","last_clean":"2024-03-23T15:57:46.377367+0000","last_became_active":"2024-03-23T15:48:04.931241+0000","last_became_peered":"2024-03-23T15:48:04.931241+0000","last_unstale":"2024-03-23T15:57:46.377367+0000","last_undegraded":"2024-03-23T15:57:46.377367+0000","last_fullsized":"2024-03-23T15:57:46.377367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362823+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:57:06.362823+0000","last_peered":"2024-03-23T15:57:06.362823+0000","last_clean":"2024-03-23T15:57:06.362823+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:57:06.362823+0000","last_undegraded":"2024-03-23T15:57:06.362823+0000","last_fullsized":"2024-03-23T15:57:06.362823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356105+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:57:06.356105+0000","last_peered":"2024-03-23T15:57:06.356105+0000","last_clean":"2024-03-23T15:57:06.356105+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:57:06.356105+0000","last_undegraded":"2024-03-23T15:57:06.356105+0000","last_fullsized":"2024-03-23T15:57:06.356105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366410+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:59:06.366410+0000","last_peered":"2024-03-23T15:59:06.366410+0000","last_clean":"2024-03-23T15:59:06.366410+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:59:06.366410+0000","last_undegraded":"2024-03-23T15:59:06.366410+0000","last_fullsized":"2024-03-23T15:59:06.366410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.305751+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:59:06.305751+0000","last_peered":"2024-03-23T15:59:06.305751+0000","last_clean":"2024-03-23T15:59:06.305751+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:59:06.305751+0000","last_undegraded":"2024-03-23T15:59:06.305751+0000","last_fullsized":"2024-03-23T15:59:06.305751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419506+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:59:06.419506+0000","last_peered":"2024-03-23T15:59:06.419506+0000","last_clean":"2024-03-23T15:59:06.419506+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:59:06.419506+0000","last_undegraded":"2024-03-23T15:59:06.419506+0000","last_fullsized":"2024-03-23T15:59:06.419506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419582+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:59:06.419582+0000","last_peered":"2024-03-23T15:59:06.419582+0000","last_clean":"2024-03-23T15:59:06.419582+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:59:06.419582+0000","last_undegraded":"2024-03-23T15:59:06.419582+0000","last_fullsized":"2024-03-23T15:59:06.419582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383777+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:59:06.383777+0000","last_peered":"2024-03-23T15:59:06.383777+0000","last_clean":"2024-03-23T15:59:06.383777+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:59:06.383777+0000","last_undegraded":"2024-03-23T15:59:06.383777+0000","last_fullsized":"2024-03-23T15:59:06.383777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417473+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:58:41.417473+0000","last_peered":"2024-03-23T15:58:41.417473+0000","last_clean":"2024-03-23T15:58:41.417473+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:58:41.417473+0000","last_undegraded":"2024-03-23T15:58:41.417473+0000","last_fullsized":"2024-03-23T15:58:41.417473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370815+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:58:41.370815+0000","last_peered":"2024-03-23T15:58:41.370815+0000","last_clean":"2024-03-23T15:58:41.370815+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:58:41.370815+0000","last_undegraded":"2024-03-23T15:58:41.370815+0000","last_fullsized":"2024-03-23T15:58:41.370815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446916+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:58:41.446916+0000","last_peered":"2024-03-23T15:58:41.446916+0000","last_clean":"2024-03-23T15:58:41.446916+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:58:41.446916+0000","last_undegraded":"2024-03-23T15:58:41.446916+0000","last_fullsized":"2024-03-23T15:58:41.446916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446527+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:58:41.446527+0000","last_peered":"2024-03-23T15:58:41.446527+0000","last_clean":"2024-03-23T15:58:41.446527+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:58:41.446527+0000","last_undegraded":"2024-03-23T15:58:41.446527+0000","last_fullsized":"2024-03-23T15:58:41.446527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303312+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:58:41.303312+0000","last_peered":"2024-03-23T15:58:41.303312+0000","last_clean":"2024-03-23T15:58:41.303312+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:58:41.303312+0000","last_undegraded":"2024-03-23T15:58:41.303312+0000","last_fullsized":"2024-03-23T15:58:41.303312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415466+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:58:16.415466+0000","last_peered":"2024-03-23T15:58:16.415466+0000","last_clean":"2024-03-23T15:58:16.415466+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:58:16.415466+0000","last_undegraded":"2024-03-23T15:58:16.415466+0000","last_fullsized":"2024-03-23T15:58:16.415466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.301061+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:58:16.301061+0000","last_peered":"2024-03-23T15:58:16.301061+0000","last_clean":"2024-03-23T15:58:16.301061+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:58:16.301061+0000","last_undegraded":"2024-03-23T15:58:16.301061+0000","last_fullsized":"2024-03-23T15:58:16.301061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.424258+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:58:16.424258+0000","last_peered":"2024-03-23T15:58:16.424258+0000","last_clean":"2024-03-23T15:58:16.424258+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:58:16.424258+0000","last_undegraded":"2024-03-23T15:58:16.424258+0000","last_fullsized":"2024-03-23T15:58:16.424258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362212+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:58:16.362212+0000","last_peered":"2024-03-23T15:58:16.362212+0000","last_clean":"2024-03-23T15:58:16.362212+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:58:16.362212+0000","last_undegraded":"2024-03-23T15:58:16.362212+0000","last_fullsized":"2024-03-23T15:58:16.362212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362193+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:58:16.362193+0000","last_peered":"2024-03-23T15:58:16.362193+0000","last_clean":"2024-03-23T15:58:16.362193+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:58:16.362193+0000","last_undegraded":"2024-03-23T15:58:16.362193+0000","last_fullsized":"2024-03-23T15:58:16.362193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298768+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:57:51.298768+0000","last_peered":"2024-03-23T15:57:51.298768+0000","last_clean":"2024-03-23T15:57:51.298768+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:57:51.298768+0000","last_undegraded":"2024-03-23T15:57:51.298768+0000","last_fullsized":"2024-03-23T15:57:51.298768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413315+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:57:51.413315+0000","last_peered":"2024-03-23T15:57:51.413315+0000","last_clean":"2024-03-23T15:57:51.413315+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:57:51.413315+0000","last_undegraded":"2024-03-23T15:57:51.413315+0000","last_fullsized":"2024-03-23T15:57:51.413315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377740+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:57:51.377740+0000","last_peered":"2024-03-23T15:57:51.377740+0000","last_clean":"2024-03-23T15:57:51.377740+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:57:51.377740+0000","last_undegraded":"2024-03-23T15:57:51.377740+0000","last_fullsized":"2024-03-23T15:57:51.377740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.422269+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:57:51.422269+0000","last_peered":"2024-03-23T15:57:51.422269+0000","last_clean":"2024-03-23T15:57:51.422269+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:57:51.422269+0000","last_undegraded":"2024-03-23T15:57:51.422269+0000","last_fullsized":"2024-03-23T15:57:51.422269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440099+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:57:26.440099+0000","last_peered":"2024-03-23T15:57:26.440099+0000","last_clean":"2024-03-23T15:57:26.440099+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:57:26.440099+0000","last_undegraded":"2024-03-23T15:57:26.440099+0000","last_fullsized":"2024-03-23T15:57:26.440099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296495+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:57:26.296495+0000","last_peered":"2024-03-23T15:57:26.296495+0000","last_clean":"2024-03-23T15:57:26.296495+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:57:26.296495+0000","last_undegraded":"2024-03-23T15:57:26.296495+0000","last_fullsized":"2024-03-23T15:57:26.296495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440553+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:57:26.440553+0000","last_peered":"2024-03-23T15:57:26.440553+0000","last_clean":"2024-03-23T15:57:26.440553+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:57:26.440553+0000","last_undegraded":"2024-03-23T15:57:26.440553+0000","last_fullsized":"2024-03-23T15:57:26.440553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.411363+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:57:26.411363+0000","last_peered":"2024-03-23T15:57:26.411363+0000","last_clean":"2024-03-23T15:57:26.411363+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:57:26.411363+0000","last_undegraded":"2024-03-23T15:57:26.411363+0000","last_fullsized":"2024-03-23T15:57:26.411363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356066+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:57:06.356066+0000","last_peered":"2024-03-23T15:57:06.356066+0000","last_clean":"2024-03-23T15:57:06.356066+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:57:06.356066+0000","last_undegraded":"2024-03-23T15:57:06.356066+0000","last_fullsized":"2024-03-23T15:57:06.356066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294507+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:57:06.294507+0000","last_peered":"2024-03-23T15:57:06.294507+0000","last_clean":"2024-03-23T15:57:06.294507+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:57:06.294507+0000","last_undegraded":"2024-03-23T15:57:06.294507+0000","last_fullsized":"2024-03-23T15:57:06.294507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.409374+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:57:06.409374+0000","last_peered":"2024-03-23T15:57:06.409374+0000","last_clean":"2024-03-23T15:57:06.409374+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:57:06.409374+0000","last_undegraded":"2024-03-23T15:57:06.409374+0000","last_fullsized":"2024-03-23T15:57:06.409374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.439044+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:57:06.439044+0000","last_peered":"2024-03-23T15:57:06.439044+0000","last_clean":"2024-03-23T15:57:06.439044+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:57:06.439044+0000","last_undegraded":"2024-03-23T15:57:06.439044+0000","last_fullsized":"2024-03-23T15:57:06.439044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.438259+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:57:06.438259+0000","last_peered":"2024-03-23T15:57:06.438259+0000","last_clean":"2024-03-23T15:57:06.438259+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:57:06.438259+0000","last_undegraded":"2024-03-23T15:57:06.438259+0000","last_fullsized":"2024-03-23T15:57:06.438259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.418592+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:57:06.418592+0000","last_peered":"2024-03-23T15:57:06.418592+0000","last_clean":"2024-03-23T15:57:06.418592+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:57:06.418592+0000","last_undegraded":"2024-03-23T15:57:06.418592+0000","last_fullsized":"2024-03-23T15:57:06.418592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362879+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:57:06.362879+0000","last_peered":"2024-03-23T15:57:06.362879+0000","last_clean":"2024-03-23T15:57:06.362879+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:57:06.362879+0000","last_undegraded":"2024-03-23T15:57:06.362879+0000","last_fullsized":"2024-03-23T15:57:06.362879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294544+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:57:06.294544+0000","last_peered":"2024-03-23T15:57:06.294544+0000","last_clean":"2024-03-23T15:57:06.294544+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:57:06.294544+0000","last_undegraded":"2024-03-23T15:57:06.294544+0000","last_fullsized":"2024-03-23T15:57:06.294544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.374127+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:57:06.374127+0000","last_peered":"2024-03-23T15:57:06.374127+0000","last_clean":"2024-03-23T15:57:06.374127+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:57:06.374127+0000","last_undegraded":"2024-03-23T15:57:06.374127+0000","last_fullsized":"2024-03-23T15:57:06.374127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419569+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:59:06.419569+0000","last_peered":"2024-03-23T15:59:06.419569+0000","last_clean":"2024-03-23T15:59:06.419569+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:59:06.419569+0000","last_undegraded":"2024-03-23T15:59:06.419569+0000","last_fullsized":"2024-03-23T15:59:06.419569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.428095+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:59:06.428095+0000","last_peered":"2024-03-23T15:59:06.428095+0000","last_clean":"2024-03-23T15:59:06.428095+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:59:06.428095+0000","last_undegraded":"2024-03-23T15:59:06.428095+0000","last_fullsized":"2024-03-23T15:59:06.428095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362853+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:57:06.362853+0000","last_peered":"2024-03-23T15:57:06.362853+0000","last_clean":"2024-03-23T15:57:06.362853+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:57:06.362853+0000","last_undegraded":"2024-03-23T15:57:06.362853+0000","last_fullsized":"2024-03-23T15:57:06.362853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383756+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:59:06.383756+0000","last_peered":"2024-03-23T15:59:06.383756+0000","last_clean":"2024-03-23T15:59:06.383756+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:59:06.383756+0000","last_undegraded":"2024-03-23T15:59:06.383756+0000","last_fullsized":"2024-03-23T15:59:06.383756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366429+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:59:06.366429+0000","last_peered":"2024-03-23T15:59:06.366429+0000","last_clean":"2024-03-23T15:59:06.366429+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:59:06.366429+0000","last_undegraded":"2024-03-23T15:59:06.366429+0000","last_fullsized":"2024-03-23T15:59:06.366429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419548+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:59:06.419548+0000","last_peered":"2024-03-23T15:59:06.419548+0000","last_clean":"2024-03-23T15:59:06.419548+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:59:06.419548+0000","last_undegraded":"2024-03-23T15:59:06.419548+0000","last_fullsized":"2024-03-23T15:59:06.419548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419558+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:59:06.419558+0000","last_peered":"2024-03-23T15:59:06.419558+0000","last_clean":"2024-03-23T15:59:06.419558+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:59:06.419558+0000","last_undegraded":"2024-03-23T15:59:06.419558+0000","last_fullsized":"2024-03-23T15:59:06.419558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.373056+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:59:06.373056+0000","last_peered":"2024-03-23T15:59:06.373056+0000","last_clean":"2024-03-23T15:59:06.373056+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:59:06.373056+0000","last_undegraded":"2024-03-23T15:59:06.373056+0000","last_fullsized":"2024-03-23T15:59:06.373056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.448540+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:59:06.448540+0000","last_peered":"2024-03-23T15:59:06.448540+0000","last_clean":"2024-03-23T15:59:06.448540+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:59:06.448540+0000","last_undegraded":"2024-03-23T15:59:06.448540+0000","last_fullsized":"2024-03-23T15:59:06.448540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.449312+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:59:06.449312+0000","last_peered":"2024-03-23T15:59:06.449312+0000","last_clean":"2024-03-23T15:59:06.449312+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:59:06.449312+0000","last_undegraded":"2024-03-23T15:59:06.449312+0000","last_fullsized":"2024-03-23T15:59:06.449312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419535+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:59:06.419535+0000","last_peered":"2024-03-23T15:59:06.419535+0000","last_clean":"2024-03-23T15:59:06.419535+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:59:06.419535+0000","last_undegraded":"2024-03-23T15:59:06.419535+0000","last_fullsized":"2024-03-23T15:59:06.419535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370893+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:58:41.370893+0000","last_peered":"2024-03-23T15:58:41.370893+0000","last_clean":"2024-03-23T15:58:41.370893+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:58:41.370893+0000","last_undegraded":"2024-03-23T15:58:41.370893+0000","last_fullsized":"2024-03-23T15:58:41.370893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.426198+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:58:41.426198+0000","last_peered":"2024-03-23T15:58:41.426198+0000","last_clean":"2024-03-23T15:58:41.426198+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:58:41.426198+0000","last_undegraded":"2024-03-23T15:58:41.426198+0000","last_fullsized":"2024-03-23T15:58:41.426198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446943+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:58:41.446943+0000","last_peered":"2024-03-23T15:58:41.446943+0000","last_clean":"2024-03-23T15:58:41.446943+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:58:41.446943+0000","last_undegraded":"2024-03-23T15:58:41.446943+0000","last_fullsized":"2024-03-23T15:58:41.446943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370879+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:58:41.370879+0000","last_peered":"2024-03-23T15:58:41.370879+0000","last_clean":"2024-03-23T15:58:41.370879+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:58:41.370879+0000","last_undegraded":"2024-03-23T15:58:41.370879+0000","last_fullsized":"2024-03-23T15:58:41.370879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303343+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:58:41.303343+0000","last_peered":"2024-03-23T15:58:41.303343+0000","last_clean":"2024-03-23T15:58:41.303343+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:58:41.303343+0000","last_undegraded":"2024-03-23T15:58:41.303343+0000","last_fullsized":"2024-03-23T15:58:41.303343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370845+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:58:41.370845+0000","last_peered":"2024-03-23T15:58:41.370845+0000","last_clean":"2024-03-23T15:58:41.370845+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:58:41.370845+0000","last_undegraded":"2024-03-23T15:58:41.370845+0000","last_fullsized":"2024-03-23T15:58:41.370845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.364338+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:58:41.364338+0000","last_peered":"2024-03-23T15:58:41.364338+0000","last_clean":"2024-03-23T15:58:41.364338+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:58:41.364338+0000","last_undegraded":"2024-03-23T15:58:41.364338+0000","last_fullsized":"2024-03-23T15:58:41.364338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417465+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:58:41.417465+0000","last_peered":"2024-03-23T15:58:41.417465+0000","last_clean":"2024-03-23T15:58:41.417465+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:58:41.417465+0000","last_undegraded":"2024-03-23T15:58:41.417465+0000","last_fullsized":"2024-03-23T15:58:41.417465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:16.449240+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:59:16.449240+0000","last_peered":"2024-03-23T15:59:16.449240+0000","last_clean":"2024-03-23T15:59:16.449240+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:59:16.449240+0000","last_undegraded":"2024-03-23T15:59:16.449240+0000","last_fullsized":"2024-03-23T15:59:16.449240+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413350+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:57:51.413350+0000","last_peered":"2024-03-23T15:57:51.413350+0000","last_clean":"2024-03-23T15:57:51.413350+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:57:51.413350+0000","last_undegraded":"2024-03-23T15:57:51.413350+0000","last_fullsized":"2024-03-23T15:57:51.413350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442763+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:57:51.442763+0000","last_peered":"2024-03-23T15:57:51.442763+0000","last_clean":"2024-03-23T15:57:51.442763+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:57:51.442763+0000","last_undegraded":"2024-03-23T15:57:51.442763+0000","last_fullsized":"2024-03-23T15:57:51.442763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.421181+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:59:26.421181+0000","last_peered":"2024-03-23T15:59:26.421181+0000","last_clean":"2024-03-23T15:59:26.421181+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:59:26.421181+0000","last_undegraded":"2024-03-23T15:59:26.421181+0000","last_fullsized":"2024-03-23T15:59:26.421181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.359976+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:57:51.359976+0000","last_peered":"2024-03-23T15:57:51.359976+0000","last_clean":"2024-03-23T15:57:51.359976+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:57:51.359976+0000","last_undegraded":"2024-03-23T15:57:51.359976+0000","last_fullsized":"2024-03-23T15:57:51.359976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377759+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:57:51.377759+0000","last_peered":"2024-03-23T15:57:51.377759+0000","last_clean":"2024-03-23T15:57:51.377759+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:57:51.377759+0000","last_undegraded":"2024-03-23T15:57:51.377759+0000","last_fullsized":"2024-03-23T15:57:51.377759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307635+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:59:26.307635+0000","last_peered":"2024-03-23T15:59:26.307635+0000","last_clean":"2024-03-23T15:59:26.307635+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:59:26.307635+0000","last_undegraded":"2024-03-23T15:59:26.307635+0000","last_fullsized":"2024-03-23T15:59:26.307635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413338+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:57:51.413338+0000","last_peered":"2024-03-23T15:57:51.413338+0000","last_clean":"2024-03-23T15:57:51.413338+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:57:51.413338+0000","last_undegraded":"2024-03-23T15:57:51.413338+0000","last_fullsized":"2024-03-23T15:57:51.413338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.366687+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:57:51.366687+0000","last_peered":"2024-03-23T15:57:51.366687+0000","last_clean":"2024-03-23T15:57:51.366687+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:57:51.366687+0000","last_undegraded":"2024-03-23T15:57:51.366687+0000","last_fullsized":"2024-03-23T15:57:51.366687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298792+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:57:51.298792+0000","last_peered":"2024-03-23T15:57:51.298792+0000","last_clean":"2024-03-23T15:57:51.298792+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:57:51.298792+0000","last_undegraded":"2024-03-23T15:57:51.298792+0000","last_fullsized":"2024-03-23T15:57:51.298792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.422291+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:57:51.422291+0000","last_peered":"2024-03-23T15:57:51.422291+0000","last_clean":"2024-03-23T15:57:51.422291+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:57:51.422291+0000","last_undegraded":"2024-03-23T15:57:51.422291+0000","last_fullsized":"2024-03-23T15:57:51.422291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413367+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:57:51.413367+0000","last_peered":"2024-03-23T15:57:51.413367+0000","last_clean":"2024-03-23T15:57:51.413367+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:57:51.413367+0000","last_undegraded":"2024-03-23T15:57:51.413367+0000","last_fullsized":"2024-03-23T15:57:51.413367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.360026+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:57:51.360026+0000","last_peered":"2024-03-23T15:57:51.360026+0000","last_clean":"2024-03-23T15:57:51.360026+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:57:51.360026+0000","last_undegraded":"2024-03-23T15:57:51.360026+0000","last_fullsized":"2024-03-23T15:57:51.360026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444935+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:58:16.444935+0000","last_peered":"2024-03-23T15:58:16.444935+0000","last_clean":"2024-03-23T15:58:16.444935+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:58:16.444935+0000","last_undegraded":"2024-03-23T15:58:16.444935+0000","last_fullsized":"2024-03-23T15:58:16.444935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444947+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:58:16.444947+0000","last_peered":"2024-03-23T15:58:16.444947+0000","last_clean":"2024-03-23T15:58:16.444947+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:58:16.444947+0000","last_undegraded":"2024-03-23T15:58:16.444947+0000","last_fullsized":"2024-03-23T15:58:16.444947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415523+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:58:16.415523+0000","last_peered":"2024-03-23T15:58:16.415523+0000","last_clean":"2024-03-23T15:58:16.415523+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:58:16.415523+0000","last_undegraded":"2024-03-23T15:58:16.415523+0000","last_fullsized":"2024-03-23T15:58:16.415523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444921+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:58:16.444921+0000","last_peered":"2024-03-23T15:58:16.444921+0000","last_clean":"2024-03-23T15:58:16.444921+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:58:16.444921+0000","last_undegraded":"2024-03-23T15:58:16.444921+0000","last_fullsized":"2024-03-23T15:58:16.444921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.368709+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:58:16.368709+0000","last_peered":"2024-03-23T15:58:16.368709+0000","last_clean":"2024-03-23T15:58:16.368709+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:58:16.368709+0000","last_undegraded":"2024-03-23T15:58:16.368709+0000","last_fullsized":"2024-03-23T15:58:16.368709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415489+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:58:16.415489+0000","last_peered":"2024-03-23T15:58:16.415489+0000","last_clean":"2024-03-23T15:58:16.415489+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:58:16.415489+0000","last_undegraded":"2024-03-23T15:58:16.415489+0000","last_fullsized":"2024-03-23T15:58:16.415489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444897+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:58:16.444897+0000","last_peered":"2024-03-23T15:58:16.444897+0000","last_clean":"2024-03-23T15:58:16.444897+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:58:16.444897+0000","last_undegraded":"2024-03-23T15:58:16.444897+0000","last_fullsized":"2024-03-23T15:58:16.444897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444865+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:58:16.444865+0000","last_peered":"2024-03-23T15:58:16.444865+0000","last_clean":"2024-03-23T15:58:16.444865+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:58:16.444865+0000","last_undegraded":"2024-03-23T15:58:16.444865+0000","last_fullsized":"2024-03-23T15:58:16.444865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362160+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:58:16.362160+0000","last_peered":"2024-03-23T15:58:16.362160+0000","last_clean":"2024-03-23T15:58:16.362160+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:58:16.362160+0000","last_undegraded":"2024-03-23T15:58:16.362160+0000","last_fullsized":"2024-03-23T15:58:16.362160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.381785+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:58:41.381785+0000","last_peered":"2024-03-23T15:58:41.381785+0000","last_clean":"2024-03-23T15:58:41.381785+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:58:41.381785+0000","last_undegraded":"2024-03-23T15:58:41.381785+0000","last_fullsized":"2024-03-23T15:58:41.381785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417439+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:58:41.417439+0000","last_peered":"2024-03-23T15:58:41.417439+0000","last_clean":"2024-03-23T15:58:41.417439+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:58:41.417439+0000","last_undegraded":"2024-03-23T15:58:41.417439+0000","last_fullsized":"2024-03-23T15:58:41.417439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415498+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:58:16.415498+0000","last_peered":"2024-03-23T15:58:16.415498+0000","last_clean":"2024-03-23T15:58:16.415498+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:58:16.415498+0000","last_undegraded":"2024-03-23T15:58:16.415498+0000","last_fullsized":"2024-03-23T15:58:16.415498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.411385+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:57:26.411385+0000","last_peered":"2024-03-23T15:57:26.411385+0000","last_clean":"2024-03-23T15:57:26.411385+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:57:26.411385+0000","last_undegraded":"2024-03-23T15:57:26.411385+0000","last_fullsized":"2024-03-23T15:57:26.411385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364494+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:57:26.364494+0000","last_peered":"2024-03-23T15:57:26.364494+0000","last_clean":"2024-03-23T15:57:26.364494+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:57:26.364494+0000","last_undegraded":"2024-03-23T15:57:26.364494+0000","last_fullsized":"2024-03-23T15:57:26.364494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440130+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:57:26.440130+0000","last_peered":"2024-03-23T15:57:26.440130+0000","last_clean":"2024-03-23T15:57:26.440130+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:57:26.440130+0000","last_undegraded":"2024-03-23T15:57:26.440130+0000","last_fullsized":"2024-03-23T15:57:26.440130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364504+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:57:26.364504+0000","last_peered":"2024-03-23T15:57:26.364504+0000","last_clean":"2024-03-23T15:57:26.364504+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:57:26.364504+0000","last_undegraded":"2024-03-23T15:57:26.364504+0000","last_fullsized":"2024-03-23T15:57:26.364504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298841+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:57:51.298841+0000","last_peered":"2024-03-23T15:57:51.298841+0000","last_clean":"2024-03-23T15:57:51.298841+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:57:51.298841+0000","last_undegraded":"2024-03-23T15:57:51.298841+0000","last_fullsized":"2024-03-23T15:57:51.298841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307684+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:59:26.307684+0000","last_peered":"2024-03-23T15:59:26.307684+0000","last_clean":"2024-03-23T15:59:26.307684+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:59:26.307684+0000","last_undegraded":"2024-03-23T15:59:26.307684+0000","last_fullsized":"2024-03-23T15:59:26.307684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307673+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:59:26.307673+0000","last_peered":"2024-03-23T15:59:26.307673+0000","last_clean":"2024-03-23T15:59:26.307673+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:59:26.307673+0000","last_undegraded":"2024-03-23T15:59:26.307673+0000","last_fullsized":"2024-03-23T15:59:26.307673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442806+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:57:51.442806+0000","last_peered":"2024-03-23T15:57:51.442806+0000","last_clean":"2024-03-23T15:57:51.442806+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:57:51.442806+0000","last_undegraded":"2024-03-23T15:57:51.442806+0000","last_fullsized":"2024-03-23T15:57:51.442806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575023,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":290816,"data_stored":95068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575022,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":96394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":685208,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":311296,"data_stored":97464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575023,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":95694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":685796,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":96556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T16:01:30.122 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T16:01:30.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T16:01:30.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.259+0000 7f7bed7e2640 1 Processor -- start 2024-03-23T16:01:30.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.260+0000 7f7bed7e2640 1 -- start start 2024-03-23T16:01:30.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.260+0000 7f7bed7e2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7be8154270 0x7f7be80ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.260+0000 7f7bed7e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be806c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.260+0000 7f7bed7e2640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7be815d880 con 0x7f7be806c650 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.260+0000 7f7bed7e2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7be80c4ae0 con 0x7f7be8154270 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.260+0000 7f7bed7e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7be80cd5e0 con 0x7f7be815ad10 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.261+0000 7f7be6ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7be8154270 0x7f7be80ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.261+0000 7f7be67fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.261+0000 7f7be77fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7be806c650 0x7f7be80c4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:37740/0 (socket says 172.21.15.129:37740) 2024-03-23T16:01:30.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.261+0000 7f7be77fe640 1 -- 172.21.15.129:0/865349211 learned_addr learned my addr 172.21.15.129:0/865349211 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:30.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.264+0000 7f7be5ffb640 1 -- 172.21.15.129:0/865349211 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190834303 0 0) 0x7f7be815d880 con 0x7f7be806c650 2024-03-23T16:01:30.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.264+0000 7f7be5ffb640 1 -- 172.21.15.129:0/865349211 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd0003520 con 0x7f7be806c650 2024-03-23T16:01:30.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.265+0000 7f7be6ffd640 1 -- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 msgr2=0x7f7be806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:30.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.265+0000 7f7be6ffd640 1 --2- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be806c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.265+0000 7f7be6ffd640 1 -- 172.21.15.129:0/865349211 >> v1:172.21.15.129:6789/0 conn(0x7f7be806c650 legacy=0x7f7be80c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.265+0000 7f7be6ffd640 1 -- 172.21.15.129:0/865349211 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be815dab0 con 0x7f7be8154270 2024-03-23T16:01:30.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.267+0000 7f7be6ffd640 1 --2- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be80ca580 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80097a0 tx=0x7f7bd802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f90fbadd8b2f2e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:30.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.272+0000 7f7be5ffb640 1 -- 172.21.15.129:0/865349211 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd802f050 con 0x7f7be8154270 2024-03-23T16:01:30.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.272+0000 7f7be5ffb640 1 -- 172.21.15.129:0/865349211 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd8031700 con 0x7f7be8154270 2024-03-23T16:01:30.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.272+0000 7f7be5ffb640 1 -- 172.21.15.129:0/865349211 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8031a90 con 0x7f7be8154270 2024-03-23T16:01:30.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 -- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 msgr2=0x7f7be80ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be80ca580 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80097a0 tx=0x7f7bd802b7a0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 -- 172.21.15.129:0/865349211 shutdown_connections 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be806c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/865349211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be80ca580 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 -- 172.21.15.129:0/865349211 >> 172.21.15.129:0/865349211 conn(0x7f7be806d290 msgr2=0x7f7be8072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 -- 172.21.15.129:0/865349211 shutdown_connections 2024-03-23T16:01:30.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.273+0000 7f7bed7e2640 1 -- 172.21.15.129:0/865349211 wait complete. 2024-03-23T16:01:30.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.275+0000 7f7bed7e2640 1 Processor -- start 2024-03-23T16:01:30.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.275+0000 7f7bed7e2640 1 -- start start 2024-03-23T16:01:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.276+0000 7f7bed7e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7be806c650 0x7f7be81f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.276+0000 7f7bed7e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.276+0000 7f7bed7e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be815cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.276+0000 7f7bed7e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7be81efe30 con 0x7f7be806c650 2024-03-23T16:01:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.276+0000 7f7bed7e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7be81e3690 con 0x7f7be8154270 2024-03-23T16:01:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.276+0000 7f7bed7e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7be81e44b0 con 0x7f7be815ad10 2024-03-23T16:01:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.277+0000 7f7be6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7be806c650 0x7f7be81f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.277+0000 7f7be77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be815cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.277+0000 7f7be67fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.277+0000 7f7be6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7be806c650 0x7f7be81f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36980/0 (socket says 172.21.15.129:36980) 2024-03-23T16:01:30.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.277+0000 7f7be6ffd640 1 -- 172.21.15.129:0/2887650172 learned_addr learned my addr 172.21.15.129:0/2887650172 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.281+0000 7f7be77fe640 1 -- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 msgr2=0x7f7be815c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.281+0000 7f7be77fe640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be815c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.281+0000 7f7be77fe640 1 -- 172.21.15.129:0/2887650172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7be806c650 msgr2=0x7f7be81f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.281+0000 7f7be77fe640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7be806c650 0x7f7be81f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.281+0000 7f7be77fe640 1 -- 172.21.15.129:0/2887650172 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd8009400 con 0x7f7be815ad10 2024-03-23T16:01:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.284+0000 7f7be77fe640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be815cd20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7be000caa0 tx=0x7f7be000cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:30.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.288+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be000ab40 con 0x7f7be815ad10 2024-03-23T16:01:30.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.288+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be000aca0 con 0x7f7be815ad10 2024-03-23T16:01:30.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.288+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0012600 con 0x7f7be815ad10 2024-03-23T16:01:30.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.288+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be81f0020 con 0x7f7be815ad10 2024-03-23T16:01:30.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.288+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be81e3b80 con 0x7f7be815ad10 2024-03-23T16:01:30.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.289+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be81e45f0 con 0x7f7be815ad10 2024-03-23T16:01:30.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.544+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7be0007070 con 0x7f7be815ad10 2024-03-23T16:01:30.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.545+0000 7f7bcf7fe640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7bd406b890 0x7f7bd406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.545+0000 7f7be6ffd640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7bd406b890 0x7f7bd406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.545+0000 7f7be6ffd640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7bd406b890 0x7f7bd406dd50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7bd802b7e0 tx=0x7f7bd8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:30.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.549+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7be0058bb0 con 0x7f7be815ad10 2024-03-23T16:01:30.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.561+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7be0054e50 con 0x7f7be815ad10 2024-03-23T16:01:30.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.776+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7be815e330 con 0x7f7bd406b890 2024-03-23T16:01:30.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.788+0000 7f7bcf7fe640 1 -- 172.21.15.129:0/2887650172 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251995 (secure 0 0 0) 0x7f7bd8041070 con 0x7f7bd406b890 2024-03-23T16:01:30.795 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:30.797 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T16:01:30.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7bd406b890 msgr2=0x7f7bd406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7bd406b890 0x7f7bd406dd50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7bd802b7e0 tx=0x7f7bd8008030 comp rx=0 tx=0).stop 2024-03-23T16:01:30.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 msgr2=0x7f7be815cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be815cd20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7be000caa0 tx=0x7f7be000cea0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 shutdown_connections 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7be815ad10 0x7f7be815cd20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7bd406b890 0x7f7bd406dd50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7be8154270 0x7f7be815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 --2- 172.21.15.129:0/2887650172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7be806c650 0x7f7be81f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 >> 172.21.15.129:0/2887650172 conn(0x7f7be806d290 msgr2=0x7f7be80c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 shutdown_connections 2024-03-23T16:01:30.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.792+0000 7f7bed7e2640 1 -- 172.21.15.129:0/2887650172 wait complete. 2024-03-23T16:01:30.805 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-03-23T16:01:30.583788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235168,"kb_used_data":2368,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739552,"statfs":{"total":773094113280,"available":772853301248,"internally_reserved":0,"allocated":2424832,"data_stored":1948916,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002675"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.366717+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:57:51.366717+0000","last_peered":"2024-03-23T15:57:51.366717+0000","last_clean":"2024-03-23T15:57:51.366717+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:57:51.366717+0000","last_undegraded":"2024-03-23T15:57:51.366717+0000","last_fullsized":"2024-03-23T15:57:51.366717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298822+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:57:51.298822+0000","last_peered":"2024-03-23T15:57:51.298822+0000","last_clean":"2024-03-23T15:57:51.298822+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:57:51.298822+0000","last_undegraded":"2024-03-23T15:57:51.298822+0000","last_fullsized":"2024-03-23T15:57:51.298822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298806+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:57:51.298806+0000","last_peered":"2024-03-23T15:57:51.298806+0000","last_clean":"2024-03-23T15:57:51.298806+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:57:51.298806+0000","last_undegraded":"2024-03-23T15:57:51.298806+0000","last_fullsized":"2024-03-23T15:57:51.298806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.429578+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T15:59:26.429578+0000","last_peered":"2024-03-23T15:59:26.429578+0000","last_clean":"2024-03-23T15:59:26.429578+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T15:59:26.429578+0000","last_undegraded":"2024-03-23T15:59:26.429578+0000","last_fullsized":"2024-03-23T15:59:26.429578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364479+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:57:26.364479+0000","last_peered":"2024-03-23T15:57:26.364479+0000","last_clean":"2024-03-23T15:57:26.364479+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:57:26.364479+0000","last_undegraded":"2024-03-23T15:57:26.364479+0000","last_fullsized":"2024-03-23T15:57:26.364479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.420429+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:57:26.420429+0000","last_peered":"2024-03-23T15:57:26.420429+0000","last_clean":"2024-03-23T15:57:26.420429+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:57:26.420429+0000","last_undegraded":"2024-03-23T15:57:26.420429+0000","last_fullsized":"2024-03-23T15:57:26.420429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364457+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:57:26.364457+0000","last_peered":"2024-03-23T15:57:26.364457+0000","last_clean":"2024-03-23T15:57:26.364457+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:57:26.364457+0000","last_undegraded":"2024-03-23T15:57:26.364457+0000","last_fullsized":"2024-03-23T15:57:26.364457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296437+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:57:26.296437+0000","last_peered":"2024-03-23T15:57:26.296437+0000","last_clean":"2024-03-23T15:57:26.296437+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:57:26.296437+0000","last_undegraded":"2024-03-23T15:57:26.296437+0000","last_fullsized":"2024-03-23T15:57:26.296437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.438228+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:57:06.438228+0000","last_peered":"2024-03-23T15:57:06.438228+0000","last_clean":"2024-03-23T15:57:06.438228+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:57:06.438228+0000","last_undegraded":"2024-03-23T15:57:06.438228+0000","last_fullsized":"2024-03-23T15:57:06.438228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362867+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:57:06.362867+0000","last_peered":"2024-03-23T15:57:06.362867+0000","last_clean":"2024-03-23T15:57:06.362867+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:57:06.362867+0000","last_undegraded":"2024-03-23T15:57:06.362867+0000","last_fullsized":"2024-03-23T15:57:06.362867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:46.377367+0000","last_change":"2024-03-23T15:48:04.931538+0000","last_active":"2024-03-23T15:57:46.377367+0000","last_peered":"2024-03-23T15:57:46.377367+0000","last_clean":"2024-03-23T15:57:46.377367+0000","last_became_active":"2024-03-23T15:48:04.931241+0000","last_became_peered":"2024-03-23T15:48:04.931241+0000","last_unstale":"2024-03-23T15:57:46.377367+0000","last_undegraded":"2024-03-23T15:57:46.377367+0000","last_fullsized":"2024-03-23T15:57:46.377367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362823+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:57:06.362823+0000","last_peered":"2024-03-23T15:57:06.362823+0000","last_clean":"2024-03-23T15:57:06.362823+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:57:06.362823+0000","last_undegraded":"2024-03-23T15:57:06.362823+0000","last_fullsized":"2024-03-23T15:57:06.362823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356105+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:57:06.356105+0000","last_peered":"2024-03-23T15:57:06.356105+0000","last_clean":"2024-03-23T15:57:06.356105+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:57:06.356105+0000","last_undegraded":"2024-03-23T15:57:06.356105+0000","last_fullsized":"2024-03-23T15:57:06.356105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366410+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:59:06.366410+0000","last_peered":"2024-03-23T15:59:06.366410+0000","last_clean":"2024-03-23T15:59:06.366410+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:59:06.366410+0000","last_undegraded":"2024-03-23T15:59:06.366410+0000","last_fullsized":"2024-03-23T15:59:06.366410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.305751+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:59:06.305751+0000","last_peered":"2024-03-23T15:59:06.305751+0000","last_clean":"2024-03-23T15:59:06.305751+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:59:06.305751+0000","last_undegraded":"2024-03-23T15:59:06.305751+0000","last_fullsized":"2024-03-23T15:59:06.305751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419506+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T15:59:06.419506+0000","last_peered":"2024-03-23T15:59:06.419506+0000","last_clean":"2024-03-23T15:59:06.419506+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T15:59:06.419506+0000","last_undegraded":"2024-03-23T15:59:06.419506+0000","last_fullsized":"2024-03-23T15:59:06.419506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419582+0000","last_change":"2024-03-23T15:47:27.451258+0000","last_active":"2024-03-23T15:59:06.419582+0000","last_peered":"2024-03-23T15:59:06.419582+0000","last_clean":"2024-03-23T15:59:06.419582+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T15:59:06.419582+0000","last_undegraded":"2024-03-23T15:59:06.419582+0000","last_fullsized":"2024-03-23T15:59:06.419582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:05:15.258284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383777+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:59:06.383777+0000","last_peered":"2024-03-23T15:59:06.383777+0000","last_clean":"2024-03-23T15:59:06.383777+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:59:06.383777+0000","last_undegraded":"2024-03-23T15:59:06.383777+0000","last_fullsized":"2024-03-23T15:59:06.383777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417473+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T15:58:41.417473+0000","last_peered":"2024-03-23T15:58:41.417473+0000","last_clean":"2024-03-23T15:58:41.417473+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T15:58:41.417473+0000","last_undegraded":"2024-03-23T15:58:41.417473+0000","last_fullsized":"2024-03-23T15:58:41.417473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:10:51.195762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370815+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T15:58:41.370815+0000","last_peered":"2024-03-23T15:58:41.370815+0000","last_clean":"2024-03-23T15:58:41.370815+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T15:58:41.370815+0000","last_undegraded":"2024-03-23T15:58:41.370815+0000","last_fullsized":"2024-03-23T15:58:41.370815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446916+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:58:41.446916+0000","last_peered":"2024-03-23T15:58:41.446916+0000","last_clean":"2024-03-23T15:58:41.446916+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:58:41.446916+0000","last_undegraded":"2024-03-23T15:58:41.446916+0000","last_fullsized":"2024-03-23T15:58:41.446916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446527+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T15:58:41.446527+0000","last_peered":"2024-03-23T15:58:41.446527+0000","last_clean":"2024-03-23T15:58:41.446527+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T15:58:41.446527+0000","last_undegraded":"2024-03-23T15:58:41.446527+0000","last_fullsized":"2024-03-23T15:58:41.446527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:33.152618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303312+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:58:41.303312+0000","last_peered":"2024-03-23T15:58:41.303312+0000","last_clean":"2024-03-23T15:58:41.303312+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:58:41.303312+0000","last_undegraded":"2024-03-23T15:58:41.303312+0000","last_fullsized":"2024-03-23T15:58:41.303312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415466+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T15:58:16.415466+0000","last_peered":"2024-03-23T15:58:16.415466+0000","last_clean":"2024-03-23T15:58:16.415466+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T15:58:16.415466+0000","last_undegraded":"2024-03-23T15:58:16.415466+0000","last_fullsized":"2024-03-23T15:58:16.415466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:51:49.645752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.301061+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:58:16.301061+0000","last_peered":"2024-03-23T15:58:16.301061+0000","last_clean":"2024-03-23T15:58:16.301061+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:58:16.301061+0000","last_undegraded":"2024-03-23T15:58:16.301061+0000","last_fullsized":"2024-03-23T15:58:16.301061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.424258+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T15:58:16.424258+0000","last_peered":"2024-03-23T15:58:16.424258+0000","last_clean":"2024-03-23T15:58:16.424258+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T15:58:16.424258+0000","last_undegraded":"2024-03-23T15:58:16.424258+0000","last_fullsized":"2024-03-23T15:58:16.424258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362212+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:58:16.362212+0000","last_peered":"2024-03-23T15:58:16.362212+0000","last_clean":"2024-03-23T15:58:16.362212+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:58:16.362212+0000","last_undegraded":"2024-03-23T15:58:16.362212+0000","last_fullsized":"2024-03-23T15:58:16.362212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362193+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:58:16.362193+0000","last_peered":"2024-03-23T15:58:16.362193+0000","last_clean":"2024-03-23T15:58:16.362193+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:58:16.362193+0000","last_undegraded":"2024-03-23T15:58:16.362193+0000","last_fullsized":"2024-03-23T15:58:16.362193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298768+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:57:51.298768+0000","last_peered":"2024-03-23T15:57:51.298768+0000","last_clean":"2024-03-23T15:57:51.298768+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:57:51.298768+0000","last_undegraded":"2024-03-23T15:57:51.298768+0000","last_fullsized":"2024-03-23T15:57:51.298768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413315+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:57:51.413315+0000","last_peered":"2024-03-23T15:57:51.413315+0000","last_clean":"2024-03-23T15:57:51.413315+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:57:51.413315+0000","last_undegraded":"2024-03-23T15:57:51.413315+0000","last_fullsized":"2024-03-23T15:57:51.413315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:02:04.929016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377740+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:57:51.377740+0000","last_peered":"2024-03-23T15:57:51.377740+0000","last_clean":"2024-03-23T15:57:51.377740+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:57:51.377740+0000","last_undegraded":"2024-03-23T15:57:51.377740+0000","last_fullsized":"2024-03-23T15:57:51.377740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.422269+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:57:51.422269+0000","last_peered":"2024-03-23T15:57:51.422269+0000","last_clean":"2024-03-23T15:57:51.422269+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:57:51.422269+0000","last_undegraded":"2024-03-23T15:57:51.422269+0000","last_fullsized":"2024-03-23T15:57:51.422269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440099+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:57:26.440099+0000","last_peered":"2024-03-23T15:57:26.440099+0000","last_clean":"2024-03-23T15:57:26.440099+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:57:26.440099+0000","last_undegraded":"2024-03-23T15:57:26.440099+0000","last_fullsized":"2024-03-23T15:57:26.440099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296495+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:57:26.296495+0000","last_peered":"2024-03-23T15:57:26.296495+0000","last_clean":"2024-03-23T15:57:26.296495+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:57:26.296495+0000","last_undegraded":"2024-03-23T15:57:26.296495+0000","last_fullsized":"2024-03-23T15:57:26.296495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440553+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:57:26.440553+0000","last_peered":"2024-03-23T15:57:26.440553+0000","last_clean":"2024-03-23T15:57:26.440553+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:57:26.440553+0000","last_undegraded":"2024-03-23T15:57:26.440553+0000","last_fullsized":"2024-03-23T15:57:26.440553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.411363+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:57:26.411363+0000","last_peered":"2024-03-23T15:57:26.411363+0000","last_clean":"2024-03-23T15:57:26.411363+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:57:26.411363+0000","last_undegraded":"2024-03-23T15:57:26.411363+0000","last_fullsized":"2024-03-23T15:57:26.411363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:32.017808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356066+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:57:06.356066+0000","last_peered":"2024-03-23T15:57:06.356066+0000","last_clean":"2024-03-23T15:57:06.356066+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:57:06.356066+0000","last_undegraded":"2024-03-23T15:57:06.356066+0000","last_fullsized":"2024-03-23T15:57:06.356066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294507+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:57:06.294507+0000","last_peered":"2024-03-23T15:57:06.294507+0000","last_clean":"2024-03-23T15:57:06.294507+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:57:06.294507+0000","last_undegraded":"2024-03-23T15:57:06.294507+0000","last_fullsized":"2024-03-23T15:57:06.294507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.409374+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:57:06.409374+0000","last_peered":"2024-03-23T15:57:06.409374+0000","last_clean":"2024-03-23T15:57:06.409374+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:57:06.409374+0000","last_undegraded":"2024-03-23T15:57:06.409374+0000","last_fullsized":"2024-03-23T15:57:06.409374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.439044+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:57:06.439044+0000","last_peered":"2024-03-23T15:57:06.439044+0000","last_clean":"2024-03-23T15:57:06.439044+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:57:06.439044+0000","last_undegraded":"2024-03-23T15:57:06.439044+0000","last_fullsized":"2024-03-23T15:57:06.439044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.438259+0000","last_change":"2024-03-23T15:47:26.149815+0000","last_active":"2024-03-23T15:57:06.438259+0000","last_peered":"2024-03-23T15:57:06.438259+0000","last_clean":"2024-03-23T15:57:06.438259+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T15:57:06.438259+0000","last_undegraded":"2024-03-23T15:57:06.438259+0000","last_fullsized":"2024-03-23T15:57:06.438259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.418592+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:57:06.418592+0000","last_peered":"2024-03-23T15:57:06.418592+0000","last_clean":"2024-03-23T15:57:06.418592+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:57:06.418592+0000","last_undegraded":"2024-03-23T15:57:06.418592+0000","last_fullsized":"2024-03-23T15:57:06.418592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362879+0000","last_change":"2024-03-23T15:47:26.153368+0000","last_active":"2024-03-23T15:57:06.362879+0000","last_peered":"2024-03-23T15:57:06.362879+0000","last_clean":"2024-03-23T15:57:06.362879+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T15:57:06.362879+0000","last_undegraded":"2024-03-23T15:57:06.362879+0000","last_fullsized":"2024-03-23T15:57:06.362879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294544+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:57:06.294544+0000","last_peered":"2024-03-23T15:57:06.294544+0000","last_clean":"2024-03-23T15:57:06.294544+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:57:06.294544+0000","last_undegraded":"2024-03-23T15:57:06.294544+0000","last_fullsized":"2024-03-23T15:57:06.294544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.374127+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:57:06.374127+0000","last_peered":"2024-03-23T15:57:06.374127+0000","last_clean":"2024-03-23T15:57:06.374127+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:57:06.374127+0000","last_undegraded":"2024-03-23T15:57:06.374127+0000","last_fullsized":"2024-03-23T15:57:06.374127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419569+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T15:59:06.419569+0000","last_peered":"2024-03-23T15:59:06.419569+0000","last_clean":"2024-03-23T15:59:06.419569+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T15:59:06.419569+0000","last_undegraded":"2024-03-23T15:59:06.419569+0000","last_fullsized":"2024-03-23T15:59:06.419569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.428095+0000","last_change":"2024-03-23T15:47:26.147196+0000","last_active":"2024-03-23T15:59:06.428095+0000","last_peered":"2024-03-23T15:59:06.428095+0000","last_clean":"2024-03-23T15:59:06.428095+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T15:59:06.428095+0000","last_undegraded":"2024-03-23T15:59:06.428095+0000","last_fullsized":"2024-03-23T15:59:06.428095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:44:26.308442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.362853+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:57:06.362853+0000","last_peered":"2024-03-23T15:57:06.362853+0000","last_clean":"2024-03-23T15:57:06.362853+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:57:06.362853+0000","last_undegraded":"2024-03-23T15:57:06.362853+0000","last_fullsized":"2024-03-23T15:57:06.362853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383756+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:59:06.383756+0000","last_peered":"2024-03-23T15:59:06.383756+0000","last_clean":"2024-03-23T15:59:06.383756+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:59:06.383756+0000","last_undegraded":"2024-03-23T15:59:06.383756+0000","last_fullsized":"2024-03-23T15:59:06.383756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366429+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:59:06.366429+0000","last_peered":"2024-03-23T15:59:06.366429+0000","last_clean":"2024-03-23T15:59:06.366429+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:59:06.366429+0000","last_undegraded":"2024-03-23T15:59:06.366429+0000","last_fullsized":"2024-03-23T15:59:06.366429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419548+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T15:59:06.419548+0000","last_peered":"2024-03-23T15:59:06.419548+0000","last_clean":"2024-03-23T15:59:06.419548+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T15:59:06.419548+0000","last_undegraded":"2024-03-23T15:59:06.419548+0000","last_fullsized":"2024-03-23T15:59:06.419548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:36.743794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419558+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T15:59:06.419558+0000","last_peered":"2024-03-23T15:59:06.419558+0000","last_clean":"2024-03-23T15:59:06.419558+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T15:59:06.419558+0000","last_undegraded":"2024-03-23T15:59:06.419558+0000","last_fullsized":"2024-03-23T15:59:06.419558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:32:39.716852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.373056+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T15:59:06.373056+0000","last_peered":"2024-03-23T15:59:06.373056+0000","last_clean":"2024-03-23T15:59:06.373056+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T15:59:06.373056+0000","last_undegraded":"2024-03-23T15:59:06.373056+0000","last_fullsized":"2024-03-23T15:59:06.373056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.448540+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T15:59:06.448540+0000","last_peered":"2024-03-23T15:59:06.448540+0000","last_clean":"2024-03-23T15:59:06.448540+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T15:59:06.448540+0000","last_undegraded":"2024-03-23T15:59:06.448540+0000","last_fullsized":"2024-03-23T15:59:06.448540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.449312+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:59:06.449312+0000","last_peered":"2024-03-23T15:59:06.449312+0000","last_clean":"2024-03-23T15:59:06.449312+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:59:06.449312+0000","last_undegraded":"2024-03-23T15:59:06.449312+0000","last_fullsized":"2024-03-23T15:59:06.449312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.419535+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T15:59:06.419535+0000","last_peered":"2024-03-23T15:59:06.419535+0000","last_clean":"2024-03-23T15:59:06.419535+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T15:59:06.419535+0000","last_undegraded":"2024-03-23T15:59:06.419535+0000","last_fullsized":"2024-03-23T15:59:06.419535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370893+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T15:58:41.370893+0000","last_peered":"2024-03-23T15:58:41.370893+0000","last_clean":"2024-03-23T15:58:41.370893+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T15:58:41.370893+0000","last_undegraded":"2024-03-23T15:58:41.370893+0000","last_fullsized":"2024-03-23T15:58:41.370893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.426198+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T15:58:41.426198+0000","last_peered":"2024-03-23T15:58:41.426198+0000","last_clean":"2024-03-23T15:58:41.426198+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T15:58:41.426198+0000","last_undegraded":"2024-03-23T15:58:41.426198+0000","last_fullsized":"2024-03-23T15:58:41.426198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446943+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:58:41.446943+0000","last_peered":"2024-03-23T15:58:41.446943+0000","last_clean":"2024-03-23T15:58:41.446943+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:58:41.446943+0000","last_undegraded":"2024-03-23T15:58:41.446943+0000","last_fullsized":"2024-03-23T15:58:41.446943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370879+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T15:58:41.370879+0000","last_peered":"2024-03-23T15:58:41.370879+0000","last_clean":"2024-03-23T15:58:41.370879+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T15:58:41.370879+0000","last_undegraded":"2024-03-23T15:58:41.370879+0000","last_fullsized":"2024-03-23T15:58:41.370879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303343+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:58:41.303343+0000","last_peered":"2024-03-23T15:58:41.303343+0000","last_clean":"2024-03-23T15:58:41.303343+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:58:41.303343+0000","last_undegraded":"2024-03-23T15:58:41.303343+0000","last_fullsized":"2024-03-23T15:58:41.303343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.370845+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T15:58:41.370845+0000","last_peered":"2024-03-23T15:58:41.370845+0000","last_clean":"2024-03-23T15:58:41.370845+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T15:58:41.370845+0000","last_undegraded":"2024-03-23T15:58:41.370845+0000","last_fullsized":"2024-03-23T15:58:41.370845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.364338+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:58:41.364338+0000","last_peered":"2024-03-23T15:58:41.364338+0000","last_clean":"2024-03-23T15:58:41.364338+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:58:41.364338+0000","last_undegraded":"2024-03-23T15:58:41.364338+0000","last_fullsized":"2024-03-23T15:58:41.364338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417465+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T15:58:41.417465+0000","last_peered":"2024-03-23T15:58:41.417465+0000","last_clean":"2024-03-23T15:58:41.417465+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T15:58:41.417465+0000","last_undegraded":"2024-03-23T15:58:41.417465+0000","last_fullsized":"2024-03-23T15:58:41.417465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:58:10.172719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:16.449240+0000","last_change":"2024-03-23T15:47:14.623697+0000","last_active":"2024-03-23T15:59:16.449240+0000","last_peered":"2024-03-23T15:59:16.449240+0000","last_clean":"2024-03-23T15:59:16.449240+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T15:59:16.449240+0000","last_undegraded":"2024-03-23T15:59:16.449240+0000","last_fullsized":"2024-03-23T15:59:16.449240+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:11.669584+0000","last_clean_scrub_stamp":"2024-03-23T15:47:11.669584+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:33:02.005205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413350+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:57:51.413350+0000","last_peered":"2024-03-23T15:57:51.413350+0000","last_clean":"2024-03-23T15:57:51.413350+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:57:51.413350+0000","last_undegraded":"2024-03-23T15:57:51.413350+0000","last_fullsized":"2024-03-23T15:57:51.413350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:55.816356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442763+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:57:51.442763+0000","last_peered":"2024-03-23T15:57:51.442763+0000","last_clean":"2024-03-23T15:57:51.442763+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:57:51.442763+0000","last_undegraded":"2024-03-23T15:57:51.442763+0000","last_fullsized":"2024-03-23T15:57:51.442763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.421181+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T15:59:26.421181+0000","last_peered":"2024-03-23T15:59:26.421181+0000","last_clean":"2024-03-23T15:59:26.421181+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T15:59:26.421181+0000","last_undegraded":"2024-03-23T15:59:26.421181+0000","last_fullsized":"2024-03-23T15:59:26.421181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:00.414650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.359976+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:57:51.359976+0000","last_peered":"2024-03-23T15:57:51.359976+0000","last_clean":"2024-03-23T15:57:51.359976+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:57:51.359976+0000","last_undegraded":"2024-03-23T15:57:51.359976+0000","last_fullsized":"2024-03-23T15:57:51.359976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377759+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:57:51.377759+0000","last_peered":"2024-03-23T15:57:51.377759+0000","last_clean":"2024-03-23T15:57:51.377759+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:57:51.377759+0000","last_undegraded":"2024-03-23T15:57:51.377759+0000","last_fullsized":"2024-03-23T15:57:51.377759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307635+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:59:26.307635+0000","last_peered":"2024-03-23T15:59:26.307635+0000","last_clean":"2024-03-23T15:59:26.307635+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:59:26.307635+0000","last_undegraded":"2024-03-23T15:59:26.307635+0000","last_fullsized":"2024-03-23T15:59:26.307635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413338+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:57:51.413338+0000","last_peered":"2024-03-23T15:57:51.413338+0000","last_clean":"2024-03-23T15:57:51.413338+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:57:51.413338+0000","last_undegraded":"2024-03-23T15:57:51.413338+0000","last_fullsized":"2024-03-23T15:57:51.413338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:37.991537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.366687+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:57:51.366687+0000","last_peered":"2024-03-23T15:57:51.366687+0000","last_clean":"2024-03-23T15:57:51.366687+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:57:51.366687+0000","last_undegraded":"2024-03-23T15:57:51.366687+0000","last_fullsized":"2024-03-23T15:57:51.366687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298792+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:57:51.298792+0000","last_peered":"2024-03-23T15:57:51.298792+0000","last_clean":"2024-03-23T15:57:51.298792+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:57:51.298792+0000","last_undegraded":"2024-03-23T15:57:51.298792+0000","last_fullsized":"2024-03-23T15:57:51.298792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.422291+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:57:51.422291+0000","last_peered":"2024-03-23T15:57:51.422291+0000","last_clean":"2024-03-23T15:57:51.422291+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:57:51.422291+0000","last_undegraded":"2024-03-23T15:57:51.422291+0000","last_fullsized":"2024-03-23T15:57:51.422291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.413367+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:57:51.413367+0000","last_peered":"2024-03-23T15:57:51.413367+0000","last_clean":"2024-03-23T15:57:51.413367+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:57:51.413367+0000","last_undegraded":"2024-03-23T15:57:51.413367+0000","last_fullsized":"2024-03-23T15:57:51.413367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:49.264301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.360026+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:57:51.360026+0000","last_peered":"2024-03-23T15:57:51.360026+0000","last_clean":"2024-03-23T15:57:51.360026+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:57:51.360026+0000","last_undegraded":"2024-03-23T15:57:51.360026+0000","last_fullsized":"2024-03-23T15:57:51.360026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444935+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:58:16.444935+0000","last_peered":"2024-03-23T15:58:16.444935+0000","last_clean":"2024-03-23T15:58:16.444935+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:58:16.444935+0000","last_undegraded":"2024-03-23T15:58:16.444935+0000","last_fullsized":"2024-03-23T15:58:16.444935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444947+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:58:16.444947+0000","last_peered":"2024-03-23T15:58:16.444947+0000","last_clean":"2024-03-23T15:58:16.444947+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:58:16.444947+0000","last_undegraded":"2024-03-23T15:58:16.444947+0000","last_fullsized":"2024-03-23T15:58:16.444947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415523+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T15:58:16.415523+0000","last_peered":"2024-03-23T15:58:16.415523+0000","last_clean":"2024-03-23T15:58:16.415523+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T15:58:16.415523+0000","last_undegraded":"2024-03-23T15:58:16.415523+0000","last_fullsized":"2024-03-23T15:58:16.415523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:20.815828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444921+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:58:16.444921+0000","last_peered":"2024-03-23T15:58:16.444921+0000","last_clean":"2024-03-23T15:58:16.444921+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:58:16.444921+0000","last_undegraded":"2024-03-23T15:58:16.444921+0000","last_fullsized":"2024-03-23T15:58:16.444921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.368709+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T15:58:16.368709+0000","last_peered":"2024-03-23T15:58:16.368709+0000","last_clean":"2024-03-23T15:58:16.368709+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T15:58:16.368709+0000","last_undegraded":"2024-03-23T15:58:16.368709+0000","last_fullsized":"2024-03-23T15:58:16.368709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415489+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T15:58:16.415489+0000","last_peered":"2024-03-23T15:58:16.415489+0000","last_clean":"2024-03-23T15:58:16.415489+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T15:58:16.415489+0000","last_undegraded":"2024-03-23T15:58:16.415489+0000","last_fullsized":"2024-03-23T15:58:16.415489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:56:31.217322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444897+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:58:16.444897+0000","last_peered":"2024-03-23T15:58:16.444897+0000","last_clean":"2024-03-23T15:58:16.444897+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:58:16.444897+0000","last_undegraded":"2024-03-23T15:58:16.444897+0000","last_fullsized":"2024-03-23T15:58:16.444897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444865+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:58:16.444865+0000","last_peered":"2024-03-23T15:58:16.444865+0000","last_clean":"2024-03-23T15:58:16.444865+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:58:16.444865+0000","last_undegraded":"2024-03-23T15:58:16.444865+0000","last_fullsized":"2024-03-23T15:58:16.444865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362160+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:58:16.362160+0000","last_peered":"2024-03-23T15:58:16.362160+0000","last_clean":"2024-03-23T15:58:16.362160+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:58:16.362160+0000","last_undegraded":"2024-03-23T15:58:16.362160+0000","last_fullsized":"2024-03-23T15:58:16.362160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.381785+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:58:41.381785+0000","last_peered":"2024-03-23T15:58:41.381785+0000","last_clean":"2024-03-23T15:58:41.381785+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:58:41.381785+0000","last_undegraded":"2024-03-23T15:58:41.381785+0000","last_fullsized":"2024-03-23T15:58:41.381785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.417439+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T15:58:41.417439+0000","last_peered":"2024-03-23T15:58:41.417439+0000","last_clean":"2024-03-23T15:58:41.417439+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T15:58:41.417439+0000","last_undegraded":"2024-03-23T15:58:41.417439+0000","last_fullsized":"2024-03-23T15:58:41.417439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:40.145664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.415498+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T15:58:16.415498+0000","last_peered":"2024-03-23T15:58:16.415498+0000","last_clean":"2024-03-23T15:58:16.415498+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T15:58:16.415498+0000","last_undegraded":"2024-03-23T15:58:16.415498+0000","last_fullsized":"2024-03-23T15:58:16.415498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:10.340613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.411385+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:57:26.411385+0000","last_peered":"2024-03-23T15:57:26.411385+0000","last_clean":"2024-03-23T15:57:26.411385+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:57:26.411385+0000","last_undegraded":"2024-03-23T15:57:26.411385+0000","last_fullsized":"2024-03-23T15:57:26.411385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364494+0000","last_change":"2024-03-23T15:47:26.152838+0000","last_active":"2024-03-23T15:57:26.364494+0000","last_peered":"2024-03-23T15:57:26.364494+0000","last_clean":"2024-03-23T15:57:26.364494+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T15:57:26.364494+0000","last_undegraded":"2024-03-23T15:57:26.364494+0000","last_fullsized":"2024-03-23T15:57:26.364494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440130+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:57:26.440130+0000","last_peered":"2024-03-23T15:57:26.440130+0000","last_clean":"2024-03-23T15:57:26.440130+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:57:26.440130+0000","last_undegraded":"2024-03-23T15:57:26.440130+0000","last_fullsized":"2024-03-23T15:57:26.440130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.364504+0000","last_change":"2024-03-23T15:47:26.152111+0000","last_active":"2024-03-23T15:57:26.364504+0000","last_peered":"2024-03-23T15:57:26.364504+0000","last_clean":"2024-03-23T15:57:26.364504+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T15:57:26.364504+0000","last_undegraded":"2024-03-23T15:57:26.364504+0000","last_fullsized":"2024-03-23T15:57:26.364504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298841+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:57:51.298841+0000","last_peered":"2024-03-23T15:57:51.298841+0000","last_clean":"2024-03-23T15:57:51.298841+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:57:51.298841+0000","last_undegraded":"2024-03-23T15:57:51.298841+0000","last_fullsized":"2024-03-23T15:57:51.298841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307684+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:59:26.307684+0000","last_peered":"2024-03-23T15:59:26.307684+0000","last_clean":"2024-03-23T15:59:26.307684+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:59:26.307684+0000","last_undegraded":"2024-03-23T15:59:26.307684+0000","last_fullsized":"2024-03-23T15:59:26.307684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307673+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:59:26.307673+0000","last_peered":"2024-03-23T15:59:26.307673+0000","last_clean":"2024-03-23T15:59:26.307673+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:59:26.307673+0000","last_undegraded":"2024-03-23T15:59:26.307673+0000","last_fullsized":"2024-03-23T15:59:26.307673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442806+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:57:51.442806+0000","last_peered":"2024-03-23T15:57:51.442806+0000","last_clean":"2024-03-23T15:57:51.442806+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:57:51.442806+0000","last_undegraded":"2024-03-23T15:57:51.442806+0000","last_fullsized":"2024-03-23T15:57:51.442806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575023,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":290816,"data_stored":95068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575022,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":96394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":685208,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":311296,"data_stored":97464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575023,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575023,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":95694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":685796,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":290816,"data_stored":96556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T16:01:30.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T16:01:30.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.944+0000 7f19c2f77640 1 Processor -- start 2024-03-23T16:01:30.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.944+0000 7f19c2f77640 1 -- start start 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.945+0000 7f19c2f77640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f19bc0ccd60 0x7f19bc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.945+0000 7f19c2f77640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f19bc0ceb00 0x7f19bc06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.945+0000 7f19c2f77640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19bc15b290 con 0x7f19bc0cef40 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.945+0000 7f19c2f77640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f19bc0ce800 con 0x7f19bc0ceb00 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.945+0000 7f19c2f77640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f19bc154a70 con 0x7f19bc0ccd60 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.946+0000 7f19bbfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f19bc0ceb00 0x7f19bc06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.946+0000 7f19bbfff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f19bc0ceb00 0x7f19bc06c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33634/0 (socket says 172.21.15.129:33634) 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.946+0000 7f19bbfff640 1 -- 172.21.15.129:0/3052007721 learned_addr learned my addr 172.21.15.129:0/3052007721 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:30.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.946+0000 7f19c0d13640 1 --2- 172.21.15.129:0/3052007721 >> v2:172.21.15.157:3300/0 conn(0x7f19bc0ccd60 0x7f19bc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.948+0000 7f19bbfff640 1 -- 172.21.15.129:0/3052007721 >> v2:172.21.15.157:3300/0 conn(0x7f19bc0ccd60 msgr2=0x7f19bc15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.948+0000 7f19bbfff640 1 --2- 172.21.15.129:0/3052007721 >> v2:172.21.15.157:3300/0 conn(0x7f19bc0ccd60 0x7f19bc15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.948+0000 7f19bbfff640 1 -- 172.21.15.129:0/3052007721 >> v1:172.21.15.157:6789/0 conn(0x7f19bc0cef40 legacy=0x7f19bc15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.948+0000 7f19bbfff640 1 -- 172.21.15.129:0/3052007721 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc0cd950 con 0x7f19bc0ceb00 2024-03-23T16:01:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.951+0000 7f19bbfff640 1 --2- 172.21.15.129:0/3052007721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc06c270 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f19bc08cf40 tx=0x7f19a802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11032ce00ae1ebda server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:30.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.955+0000 7f19bb7fe640 1 -- 172.21.15.129:0/3052007721 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a802f850 con 0x7f19bc0ceb00 2024-03-23T16:01:30.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.955+0000 7f19bb7fe640 1 -- 172.21.15.129:0/3052007721 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a802f9b0 con 0x7f19bc0ceb00 2024-03-23T16:01:30.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.955+0000 7f19bb7fe640 1 -- 172.21.15.129:0/3052007721 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a802fd40 con 0x7f19bc0ceb00 2024-03-23T16:01:30.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 -- 172.21.15.129:0/3052007721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 msgr2=0x7f19bc06c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 --2- 172.21.15.129:0/3052007721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc06c270 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f19bc08cf40 tx=0x7f19a802ba30 comp rx=0 tx=0).stop 2024-03-23T16:01:30.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 -- 172.21.15.129:0/3052007721 shutdown_connections 2024-03-23T16:01:30.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 --2- 172.21.15.129:0/3052007721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc06c270 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 --2- 172.21.15.129:0/3052007721 >> v2:172.21.15.157:3300/0 conn(0x7f19bc0ccd60 0x7f19bc15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 -- 172.21.15.129:0/3052007721 >> 172.21.15.129:0/3052007721 conn(0x7f19bc0b1840 msgr2=0x7f19bc0b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:30.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 -- 172.21.15.129:0/3052007721 shutdown_connections 2024-03-23T16:01:30.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.956+0000 7f19c2f77640 1 -- 172.21.15.129:0/3052007721 wait complete. 2024-03-23T16:01:30.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.958+0000 7f19c2f77640 1 Processor -- start 2024-03-23T16:01:30.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.958+0000 7f19c2f77640 1 -- start start 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19c2f77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19bc0ccd60 0x7f19bc1f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19c2f77640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc0cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19c2f77640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19bc0cef40 0x7f19bc1f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19c2f77640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1e5980 con 0x7f19bc0ceb00 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19c2f77640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1f0b10 con 0x7f19bc0ccd60 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19c2f77640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f19bc1f0dd0 con 0x7f19bc0cef40 2024-03-23T16:01:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.959+0000 7f19bbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc0cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.960+0000 7f19c1514640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19bc0cef40 0x7f19bc1f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.960+0000 7f19c0d13640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19bc0ccd60 0x7f19bc1f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:30.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.960+0000 7f19bbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc0cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33638/0 (socket says 172.21.15.129:33638) 2024-03-23T16:01:30.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.960+0000 7f19bbfff640 1 -- 172.21.15.129:0/1892313414 learned_addr learned my addr 172.21.15.129:0/1892313414 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.963+0000 7f19bbfff640 1 -- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19bc0cef40 msgr2=0x7f19bc1f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.963+0000 7f19bbfff640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19bc0cef40 0x7f19bc1f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.963+0000 7f19bbfff640 1 -- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19bc0ccd60 msgr2=0x7f19bc1f16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:30.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.963+0000 7f19bbfff640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19bc0ccd60 0x7f19bc1f16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:30.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.963+0000 7f19bbfff640 1 -- 172.21.15.129:0/1892313414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a80094d0 con 0x7f19bc0ceb00 2024-03-23T16:01:30.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.966+0000 7f19bbfff640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc0cbb20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f19a802ba70 tx=0x7f19a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:30.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.970+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8037c40 con 0x7f19bc0ceb00 2024-03-23T16:01:30.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.970+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a8037dc0 con 0x7f19bc0ceb00 2024-03-23T16:01:30.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.970+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc1f10b0 con 0x7f19bc0ceb00 2024-03-23T16:01:30.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.970+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8041640 con 0x7f19bc0ceb00 2024-03-23T16:01:30.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.970+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc1e52c0 con 0x7f19bc0ceb00 2024-03-23T16:01:30.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:30.971+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19bc04f530 con 0x7f19bc0ceb00 2024-03-23T16:01:31.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.256+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f19a80417a0 con 0x7f19bc0ceb00 2024-03-23T16:01:31.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.256+0000 7f19b97fa640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f19a006b860 0x7f19a006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:31.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.256+0000 7f19c0d13640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f19a006b860 0x7f19a006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:31.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.257+0000 7f19c0d13640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f19a006b860 0x7f19a006dd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009aa0 tx=0x7f19b0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:31.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.279+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f19a803c070 con 0x7f19bc0ceb00 2024-03-23T16:01:31.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.279+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f19a8078530 con 0x7f19bc0ceb00 2024-03-23T16:01:31.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.465+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19bc06db50 con 0x7f19bc0ceb00 2024-03-23T16:01:31.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.537+0000 7f19b97fa640 1 -- 172.21.15.129:0/1892313414 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16825 (secure 0 0 0) 0x7f19a807bbf0 con 0x7f19bc0ceb00 2024-03-23T16:01:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:31.540 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":24,"fsid":"24ba5e35-b5b5-4c1d-a218-52a63f3af0eb","created":"2024-03-23T15:46:56.649468+0000","modified":"2024-03-23T15:48:33.918711+0000","last_up_change":"2024-03-23T15:47:10.167917+0000","last_in_change":"2024-03-23T15:47:01.927700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T15:47:11.061293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T15:47:24.638536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T15:47:25.617795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4924c046-2ce0-43d1-a4e5-d8d53ba55590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6825","nonce":1377293601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6827","nonce":1377293601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6831","nonce":1377293601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1377293601},{"type":"v1","addr":"172.21.15.129:6829","nonce":1377293601}]},"public_addr":"172.21.15.129:6825/1377293601","cluster_addr":"172.21.15.129:6827/1377293601","heartbeat_back_addr":"172.21.15.129:6831/1377293601","heartbeat_front_addr":"172.21.15.129:6829/1377293601","state":["exists","up"]},{"osd":1,"uuid":"59cec624-7f42-4a23-a75a-ee39f84bac23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6817","nonce":168045261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6819","nonce":168045261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6823","nonce":168045261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":168045261},{"type":"v1","addr":"172.21.15.129:6821","nonce":168045261}]},"public_addr":"172.21.15.129:6817/168045261","cluster_addr":"172.21.15.129:6819/168045261","heartbeat_back_addr":"172.21.15.129:6823/168045261","heartbeat_front_addr":"172.21.15.129:6821/168045261","state":["exists","up"]},{"osd":2,"uuid":"b246bd79-82a4-4808-82cc-c78d286260ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6801","nonce":2679851475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6805","nonce":2679851475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6813","nonce":2679851475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":2679851475},{"type":"v1","addr":"172.21.15.129:6809","nonce":2679851475}]},"public_addr":"172.21.15.129:6801/2679851475","cluster_addr":"172.21.15.129:6805/2679851475","heartbeat_back_addr":"172.21.15.129:6813/2679851475","heartbeat_front_addr":"172.21.15.129:6809/2679851475","state":["exists","up"]},{"osd":3,"uuid":"9041cd1c-9759-411f-bc7e-182544af5af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6804","nonce":3775172849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6808","nonce":3775172849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6815","nonce":3775172849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":3775172849},{"type":"v1","addr":"172.21.15.129:6812","nonce":3775172849}]},"public_addr":"172.21.15.129:6804/3775172849","cluster_addr":"172.21.15.129:6808/3775172849","heartbeat_back_addr":"172.21.15.129:6815/3775172849","heartbeat_front_addr":"172.21.15.129:6812/3775172849","state":["exists","up"]},{"osd":4,"uuid":"6ae79e67-6ebc-48ac-931d-5057f09c098c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6827","nonce":4230040520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6829","nonce":4230040520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6833","nonce":4230040520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":4230040520},{"type":"v1","addr":"172.21.15.157:6831","nonce":4230040520}]},"public_addr":"172.21.15.157:6827/4230040520","cluster_addr":"172.21.15.157:6829/4230040520","heartbeat_back_addr":"172.21.15.157:6833/4230040520","heartbeat_front_addr":"172.21.15.157:6831/4230040520","state":["exists","up"]},{"osd":5,"uuid":"f679eb27-e464-459b-ab03-06c91915c3be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6803","nonce":869595834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6805","nonce":869595834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6809","nonce":869595834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":869595834},{"type":"v1","addr":"172.21.15.157:6807","nonce":869595834}]},"public_addr":"172.21.15.157:6803/869595834","cluster_addr":"172.21.15.157:6805/869595834","heartbeat_back_addr":"172.21.15.157:6809/869595834","heartbeat_front_addr":"172.21.15.157:6807/869595834","state":["exists","up"]},{"osd":6,"uuid":"f4438f7e-f95e-4ae6-91aa-4cedeb8b31db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6811","nonce":4270775080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6813","nonce":4270775080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6817","nonce":4270775080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":4270775080},{"type":"v1","addr":"172.21.15.157:6815","nonce":4270775080}]},"public_addr":"172.21.15.157:6811/4270775080","cluster_addr":"172.21.15.157:6813/4270775080","heartbeat_back_addr":"172.21.15.157:6817/4270775080","heartbeat_front_addr":"172.21.15.157:6815/4270775080","state":["exists","up"]},{"osd":7,"uuid":"21ca1071-c633-4690-a97b-15ff27ea72b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6819","nonce":2526630177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6821","nonce":2526630177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6825","nonce":2526630177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2526630177},{"type":"v1","addr":"172.21.15.157:6823","nonce":2526630177}]},"public_addr":"172.21.15.157:6819/2526630177","cluster_addr":"172.21.15.157:6821/2526630177","heartbeat_back_addr":"172.21.15.157:6825/2526630177","heartbeat_front_addr":"172.21.15.157:6823/2526630177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.181671+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.190268+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.102156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.170511+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.208007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.083319+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.128358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T15:47:07.135432+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.35","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.157:6837/3877606926":"2024-03-24T15:48:33.905939+0000","172.21.15.157:6836/3877606926":"2024-03-24T15:48:33.905939+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f19a006b860 msgr2=0x7f19a006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f19a006b860 0x7f19a006dd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009aa0 tx=0x7f19b0009190 comp rx=0 tx=0).stop 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 msgr2=0x7f19bc0cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc0cbb20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f19a802ba70 tx=0x7f19a8004730 comp rx=0 tx=0).stop 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 shutdown_connections 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19bc0cef40 0x7f19bc1f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:31.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19bc0ceb00 0x7f19bc0cbb20 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:31.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f19a006b860 0x7f19a006dd20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:31.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 --2- 172.21.15.129:0/1892313414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19bc0ccd60 0x7f19bc1f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:31.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 >> 172.21.15.129:0/1892313414 conn(0x7f19bc0b1840 msgr2=0x7f19bc0b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:31.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 shutdown_connections 2024-03-23T16:01:31.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:31.541+0000 7f19c2f77640 1 -- 172.21.15.129:0/1892313414 wait complete. 2024-03-23T16:01:32.555 INFO:tasks.ceph:Scrubbing osd.0 2024-03-23T16:01:32.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:32.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.660+0000 7efe6d177640 1 Processor -- start 2024-03-23T16:01:32.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.660+0000 7efe6d177640 1 -- start start 2024-03-23T16:01:32.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.661+0000 7efe6d177640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe680cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.661+0000 7efe6d177640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe6806bb70 0x7efe680ca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.661+0000 7efe6d177640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6815a420 con 0x7efe6806dbb0 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.661+0000 7efe6d177640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7efe6806df90 con 0x7efe6806bb70 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.661+0000 7efe6d177640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efe680ccae0 con 0x7efe6815b130 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.662+0000 7efe6759e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe6806bb70 0x7efe680ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.662+0000 7efe6659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe680cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.662+0000 7efe6759e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7efe6806bb70 0x7efe680ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33642/0 (socket says 172.21.15.129:33642) 2024-03-23T16:01:32.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.662+0000 7efe6759e640 1 -- 172.21.15.129:0/4130345000 learned_addr learned my addr 172.21.15.129:0/4130345000 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:32.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.664+0000 7efe6759e640 1 -- 172.21.15.129:0/4130345000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 msgr2=0x7efe680cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:32.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.664+0000 7efe6759e640 1 --2- 172.21.15.129:0/4130345000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe680cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.664+0000 7efe6759e640 1 -- 172.21.15.129:0/4130345000 >> v1:172.21.15.157:6789/0 conn(0x7efe6806dbb0 legacy=0x7efe680cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.664+0000 7efe6759e640 1 -- 172.21.15.129:0/4130345000 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe680ccdc0 con 0x7efe6806bb70 2024-03-23T16:01:32.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.667+0000 7efe6759e640 1 --2- 172.21.15.129:0/4130345000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806bb70 0x7efe680ca530 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe60009e80 tx=0x7efe6002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ded8a473478381 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.670+0000 7efe65d9b640 1 -- 172.21.15.129:0/4130345000 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6002cb70 con 0x7efe6806bb70 2024-03-23T16:01:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.670+0000 7efe65d9b640 1 -- 172.21.15.129:0/4130345000 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6002ccd0 con 0x7efe6806bb70 2024-03-23T16:01:32.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.670+0000 7efe65d9b640 1 -- 172.21.15.129:0/4130345000 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60034610 con 0x7efe6806bb70 2024-03-23T16:01:32.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 -- 172.21.15.129:0/4130345000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806bb70 msgr2=0x7efe680ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 --2- 172.21.15.129:0/4130345000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806bb70 0x7efe680ca530 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe60009e80 tx=0x7efe6002bfd0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 -- 172.21.15.129:0/4130345000 shutdown_connections 2024-03-23T16:01:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 --2- 172.21.15.129:0/4130345000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806bb70 0x7efe680ca530 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 --2- 172.21.15.129:0/4130345000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe680cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 -- 172.21.15.129:0/4130345000 >> 172.21.15.129:0/4130345000 conn(0x7efe680cfbb0 msgr2=0x7efe680cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 -- 172.21.15.129:0/4130345000 shutdown_connections 2024-03-23T16:01:32.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.671+0000 7efe6d177640 1 -- 172.21.15.129:0/4130345000 wait complete. 2024-03-23T16:01:32.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.673+0000 7efe6d177640 1 Processor -- start 2024-03-23T16:01:32.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.673+0000 7efe6d177640 1 -- start start 2024-03-23T16:01:32.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.674+0000 7efe6d177640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe6806bb70 0x7efe681571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.674+0000 7efe6d177640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806dbb0 0x7efe681efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.674+0000 7efe6d177640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe68146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.674+0000 7efe6d177640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe6814b740 con 0x7efe6806dbb0 2024-03-23T16:01:32.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.674+0000 7efe6d177640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efe6814cbd0 con 0x7efe6806bb70 2024-03-23T16:01:32.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.674+0000 7efe6d177640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efe680b2260 con 0x7efe6815b130 2024-03-23T16:01:32.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.675+0000 7efe6659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806dbb0 0x7efe681efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.675+0000 7efe6759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe68146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.675+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe6806bb70 0x7efe681571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.675+0000 7efe6759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe68146d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:39586/0 (socket says 172.21.15.129:39586) 2024-03-23T16:01:32.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.675+0000 7efe6759e640 1 -- 172.21.15.129:0/407991776 learned_addr learned my addr 172.21.15.129:0/407991776 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:32.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.679+0000 7efe6659c640 1 -- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 msgr2=0x7efe68146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.679+0000 7efe6659c640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efe6815b130 0x7efe68146d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.679+0000 7efe6659c640 1 -- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe6806bb70 msgr2=0x7efe681571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.679+0000 7efe6659c640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe6806bb70 0x7efe681571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.679+0000 7efe6659c640 1 -- 172.21.15.129:0/407991776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe60009b30 con 0x7efe6806dbb0 2024-03-23T16:01:32.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.681+0000 7efe6659c640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806dbb0 0x7efe681efef0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00e870 tx=0x7efe5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.684+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00ee60 con 0x7efe6806dbb0 2024-03-23T16:01:32.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.684+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe5c00c940 con 0x7efe6806dbb0 2024-03-23T16:01:32.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.684+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe680cba90 con 0x7efe6806dbb0 2024-03-23T16:01:32.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.684+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00ccd0 con 0x7efe6806dbb0 2024-03-23T16:01:32.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.684+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6814b360 con 0x7efe6806dbb0 2024-03-23T16:01:32.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.685+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efe6815ac10 con 0x7efe6806dbb0 2024-03-23T16:01:32.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.706+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efe5c013470 con 0x7efe6806dbb0 2024-03-23T16:01:32.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.707+0000 7efe4f7fe640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe5406b890 0x7efe5406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.707+0000 7efe66d9d640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe5406b890 0x7efe5406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.708+0000 7efe66d9d640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe5406b890 0x7efe5406dd50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efe58001780 tx=0x7efe58009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.710+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7efe5c014e40 con 0x7efe6806dbb0 2024-03-23T16:01:32.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.711+0000 7efe4f7fe640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7efe54073b90 0x7efe54075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.711+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 --> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe54076680 con 0x7efe54073b90 2024-03-23T16:01:32.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.711+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efe5c004640 con 0x7efe6806dbb0 2024-03-23T16:01:32.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.711+0000 7efe6759e640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7efe54073b90 0x7efe54075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.711+0000 7efe6759e640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7efe54073b90 0x7efe54075fb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.712+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== osd.0 v2:172.21.15.129:6824/1377293601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efe54076680 con 0x7efe54073b90 2024-03-23T16:01:32.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.726+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 --> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7efe6815a930 con 0x7efe54073b90 2024-03-23T16:01:32.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.726+0000 7efe4f7fe640 1 -- 172.21.15.129:0/407991776 <== osd.0 v2:172.21.15.129:6824/1377293601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7efe6002ca70 con 0x7efe54073b90 2024-03-23T16:01:32.729 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:32.729 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:32.729 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7efe54073b90 msgr2=0x7efe54075fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:6824/1377293601,v1:172.21.15.129:6825/1377293601] conn(0x7efe54073b90 0x7efe54075fb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe5406b890 msgr2=0x7efe5406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7efe5406b890 0x7efe5406dd50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efe58001780 tx=0x7efe58009040 comp rx=0 tx=0).stop 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806dbb0 msgr2=0x7efe681efef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 --2- 172.21.15.129:0/407991776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe6806dbb0 0x7efe681efef0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00e870 tx=0x7efe5c00ed30 comp rx=0 tx=0).stop 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6759e640 1 -- 172.21.15.129:0/407991776 reap_dead start 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 shutdown_connections 2024-03-23T16:01:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 >> 172.21.15.129:0/407991776 conn(0x7efe680cfbb0 msgr2=0x7efe680c8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 shutdown_connections 2024-03-23T16:01:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.727+0000 7efe6d177640 1 -- 172.21.15.129:0/407991776 wait complete. 2024-03-23T16:01:32.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-23T16:01:32.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.850+0000 7fb22848f640 1 Processor -- start 2024-03-23T16:01:32.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.850+0000 7fb22848f640 1 -- start start 2024-03-23T16:01:32.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.851+0000 7fb22848f640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb2200cb780 0x7fb22015d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.851+0000 7fb22848f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb22015afb0 0x7fb2200c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.851+0000 7fb22848f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2200cc2a0 con 0x7fb220154590 2024-03-23T16:01:32.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.851+0000 7fb22848f640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb22015ab20 con 0x7fb2200cb780 2024-03-23T16:01:32.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.851+0000 7fb22848f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2200cbce0 con 0x7fb22015afb0 2024-03-23T16:01:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.895+0000 7fb22622b640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb220154590 0x7fb2200ca990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55154/0 (socket says 172.21.15.129:55154) 2024-03-23T16:01:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.895+0000 7fb22622b640 1 -- 172.21.15.129:0/14969299 learned_addr learned my addr 172.21.15.129:0/14969299 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.895+0000 7fb225a2a640 1 --2- 172.21.15.129:0/14969299 >> v2:172.21.15.129:3300/0 conn(0x7fb2200cb780 0x7fb22015d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.903+0000 7fb225229640 1 -- 172.21.15.129:0/14969299 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560631775 0 0) 0x7fb2200cc2a0 con 0x7fb220154590 2024-03-23T16:01:32.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.903+0000 7fb225229640 1 -- 172.21.15.129:0/14969299 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb20c003520 con 0x7fb220154590 2024-03-23T16:01:32.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.905+0000 7fb225a2a640 1 -- 172.21.15.129:0/14969299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb22015afb0 msgr2=0x7fb2200c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:32.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.905+0000 7fb225a2a640 1 --2- 172.21.15.129:0/14969299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb22015afb0 0x7fb2200c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.905+0000 7fb225a2a640 1 -- 172.21.15.129:0/14969299 >> v1:172.21.15.129:6789/0 conn(0x7fb220154590 legacy=0x7fb2200ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.905+0000 7fb225a2a640 1 -- 172.21.15.129:0/14969299 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2200cbfc0 con 0x7fb2200cb780 2024-03-23T16:01:32.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.907+0000 7fb225a2a640 1 --2- 172.21.15.129:0/14969299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb22015d920 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb2100097a0 tx=0x7fb21002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede093402e4816a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.912+0000 7fb225229640 1 -- 172.21.15.129:0/14969299 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21002e070 con 0x7fb2200cb780 2024-03-23T16:01:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.912+0000 7fb225229640 1 -- 172.21.15.129:0/14969299 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2100315a0 con 0x7fb2200cb780 2024-03-23T16:01:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.912+0000 7fb225229640 1 -- 172.21.15.129:0/14969299 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210031930 con 0x7fb2200cb780 2024-03-23T16:01:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 -- 172.21.15.129:0/14969299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 msgr2=0x7fb22015d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 --2- 172.21.15.129:0/14969299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb22015d920 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb2100097a0 tx=0x7fb21002b840 comp rx=0 tx=0).stop 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 -- 172.21.15.129:0/14969299 shutdown_connections 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 --2- 172.21.15.129:0/14969299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb22015afb0 0x7fb2200c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 --2- 172.21.15.129:0/14969299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb22015d920 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 -- 172.21.15.129:0/14969299 >> 172.21.15.129:0/14969299 conn(0x7fb220061b50 msgr2=0x7fb220061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 -- 172.21.15.129:0/14969299 shutdown_connections 2024-03-23T16:01:32.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.913+0000 7fb22848f640 1 -- 172.21.15.129:0/14969299 wait complete. 2024-03-23T16:01:32.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.915+0000 7fb22848f640 1 Processor -- start 2024-03-23T16:01:32.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.915+0000 7fb22848f640 1 -- start start 2024-03-23T16:01:32.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.916+0000 7fb22848f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb2200d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.916+0000 7fb22848f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb220154590 0x7fb2200c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.916+0000 7fb22848f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb22015afb0 0x7fb2200c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.916+0000 7fb22848f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2201f0990 con 0x7fb2200cb780 2024-03-23T16:01:32.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.916+0000 7fb22848f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb2200d0830 con 0x7fb22015afb0 2024-03-23T16:01:32.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.916+0000 7fb22848f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2201efd80 con 0x7fb220154590 2024-03-23T16:01:32.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.917+0000 7fb22622b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb2200d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.917+0000 7fb22622b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb2200d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33672/0 (socket says 172.21.15.129:33672) 2024-03-23T16:01:32.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.917+0000 7fb22622b640 1 -- 172.21.15.129:0/2611130049 learned_addr learned my addr 172.21.15.129:0/2611130049 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:32.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.917+0000 7fb226a2c640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb22015afb0 0x7fb2200c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.920+0000 7fb22622b640 1 -- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb220154590 msgr2=0x7fb2200c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.920+0000 7fb22622b640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb220154590 0x7fb2200c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.920+0000 7fb22622b640 1 -- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb22015afb0 msgr2=0x7fb2200c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:32.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.920+0000 7fb22622b640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb22015afb0 0x7fb2200c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:32.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.920+0000 7fb22622b640 1 -- 172.21.15.129:0/2611130049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb210009400 con 0x7fb2200cb780 2024-03-23T16:01:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.923+0000 7fb22622b640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb2200d0180 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00c430 tx=0x7fb21c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.926+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c005ce0 con 0x7fb2200cb780 2024-03-23T16:01:32.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.927+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb21c005e40 con 0x7fb2200cb780 2024-03-23T16:01:32.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.927+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2201f0060 con 0x7fb2200cb780 2024-03-23T16:01:32.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.927+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0072f0 con 0x7fb2200cb780 2024-03-23T16:01:32.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.927+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2201f0620 con 0x7fb2200cb780 2024-03-23T16:01:32.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.928+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2200757b0 con 0x7fb2200cb780 2024-03-23T16:01:32.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.947+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb21c007450 con 0x7fb2200cb780 2024-03-23T16:01:32.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.947+0000 7fb216ffd640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb20406b890 0x7fb20406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:32.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.948+0000 7fb225a2a640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb20406b890 0x7fb20406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:32.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.948+0000 7fb225a2a640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb20406b890 0x7fb20406dd50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb210036000 tx=0x7fb210030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:32.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:32.949+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb21c058510 con 0x7fb2200cb780 2024-03-23T16:01:33.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.200+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb21c0582e0 con 0x7fb2200cb780 2024-03-23T16:01:33.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.395+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fb220073350 con 0x7fb20406b890 2024-03-23T16:01:33.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.396+0000 7fb216ffd640 1 -- 172.21.15.129:0/2611130049 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb220073350 con 0x7fb20406b890 2024-03-23T16:01:33.398 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 0 to deep-scrub 2024-03-23T16:01:33.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb20406b890 msgr2=0x7fb20406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb20406b890 0x7fb20406dd50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb210036000 tx=0x7fb210030000 comp rx=0 tx=0).stop 2024-03-23T16:01:33.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 msgr2=0x7fb2200d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb2200d0180 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00c430 tx=0x7fb21c00c8f0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 shutdown_connections 2024-03-23T16:01:33.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb20406b890 0x7fb20406dd50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb22015afb0 0x7fb2200c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb220154590 0x7fb2200c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 --2- 172.21.15.129:0/2611130049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2200cb780 0x7fb2200d0180 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 >> 172.21.15.129:0/2611130049 conn(0x7fb220061b50 msgr2=0x7fb2200c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:33.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 shutdown_connections 2024-03-23T16:01:33.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.398+0000 7fb22848f640 1 -- 172.21.15.129:0/2611130049 wait complete. 2024-03-23T16:01:33.411 INFO:tasks.ceph:Scrubbing osd.1 2024-03-23T16:01:33.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:33.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.516+0000 7f3e199e4640 1 Processor -- start 2024-03-23T16:01:33.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.516+0000 7f3e199e4640 1 -- start start 2024-03-23T16:01:33.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.517+0000 7f3e199e4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e14143680 0x7f3e140cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.517+0000 7f3e199e4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3e14071210 0x7f3e14071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.517+0000 7f3e199e4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1415a630 con 0x7f3e1415d8d0 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.517+0000 7f3e199e4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3e14143a60 con 0x7f3e14071210 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.517+0000 7f3e199e4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e140c9240 con 0x7f3e14143680 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.518+0000 7f3e127fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3e1415d8d0 0x7f3e1415aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55166/0 (socket says 172.21.15.129:55166) 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.518+0000 7f3e127fc640 1 -- 172.21.15.129:0/3938835134 learned_addr learned my addr 172.21.15.129:0/3938835134 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.518+0000 7f3e137fe640 1 --2- 172.21.15.129:0/3938835134 >> v2:172.21.15.157:3300/0 conn(0x7f3e14071210 0x7f3e14071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.518+0000 7f3e12ffd640 1 --2- 172.21.15.129:0/3938835134 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e14143680 0x7f3e140cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.520+0000 7f3e11ffb640 1 -- 172.21.15.129:0/3938835134 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2552849504 0 0) 0x7f3e1415a630 con 0x7f3e1415d8d0 2024-03-23T16:01:33.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.521+0000 7f3e11ffb640 1 -- 172.21.15.129:0/3938835134 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dfc003520 con 0x7f3e1415d8d0 2024-03-23T16:01:33.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.522+0000 7f3e12ffd640 1 -- 172.21.15.129:0/3938835134 >> v2:172.21.15.157:3300/0 conn(0x7f3e14071210 msgr2=0x7f3e14071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:33.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.522+0000 7f3e12ffd640 1 --2- 172.21.15.129:0/3938835134 >> v2:172.21.15.157:3300/0 conn(0x7f3e14071210 0x7f3e14071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.522+0000 7f3e12ffd640 1 -- 172.21.15.129:0/3938835134 >> v1:172.21.15.129:6789/0 conn(0x7f3e1415d8d0 legacy=0x7f3e1415aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.522+0000 7f3e12ffd640 1 -- 172.21.15.129:0/3938835134 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1415a2e0 con 0x7f3e14143680 2024-03-23T16:01:33.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.524+0000 7f3e12ffd640 1 --2- 172.21.15.129:0/3938835134 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e14143680 0x7f3e140cabd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e040097a0 tx=0x7f3e0402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed04ec70086fbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:33.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.529+0000 7f3e11ffb640 1 -- 172.21.15.129:0/3938835134 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04030610 con 0x7f3e14143680 2024-03-23T16:01:33.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.529+0000 7f3e11ffb640 1 -- 172.21.15.129:0/3938835134 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e04030770 con 0x7f3e14143680 2024-03-23T16:01:33.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.529+0000 7f3e11ffb640 1 -- 172.21.15.129:0/3938835134 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04030b00 con 0x7f3e14143680 2024-03-23T16:01:33.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 -- 172.21.15.129:0/3938835134 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e14143680 msgr2=0x7f3e140cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 --2- 172.21.15.129:0/3938835134 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e14143680 0x7f3e140cabd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e040097a0 tx=0x7f3e0402b960 comp rx=0 tx=0).stop 2024-03-23T16:01:33.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 -- 172.21.15.129:0/3938835134 shutdown_connections 2024-03-23T16:01:33.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 --2- 172.21.15.129:0/3938835134 >> v2:172.21.15.157:3300/0 conn(0x7f3e14071210 0x7f3e14071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 --2- 172.21.15.129:0/3938835134 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e14143680 0x7f3e140cabd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 -- 172.21.15.129:0/3938835134 >> 172.21.15.129:0/3938835134 conn(0x7f3e1406b310 msgr2=0x7f3e1406f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:33.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 -- 172.21.15.129:0/3938835134 shutdown_connections 2024-03-23T16:01:33.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.530+0000 7f3e199e4640 1 -- 172.21.15.129:0/3938835134 wait complete. 2024-03-23T16:01:33.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.532+0000 7f3e199e4640 1 Processor -- start 2024-03-23T16:01:33.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.532+0000 7f3e199e4640 1 -- start start 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.533+0000 7f3e199e4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e14071210 0x7f3e141f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.533+0000 7f3e199e4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e14143680 0x7f3e140b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.533+0000 7f3e199e4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e1415d8d0 0x7f3e140cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.533+0000 7f3e199e4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e140ce230 con 0x7f3e14143680 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.533+0000 7f3e199e4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3e14140c90 con 0x7f3e14071210 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.533+0000 7f3e199e4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3e140ce760 con 0x7f3e1415d8d0 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.534+0000 7f3e127fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e14143680 0x7f3e140b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.534+0000 7f3e137fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e1415d8d0 0x7f3e140cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.534+0000 7f3e12ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e14071210 0x7f3e141f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.534+0000 7f3e127fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e14143680 0x7f3e140b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33684/0 (socket says 172.21.15.129:33684) 2024-03-23T16:01:33.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.534+0000 7f3e127fc640 1 -- 172.21.15.129:0/2598347033 learned_addr learned my addr 172.21.15.129:0/2598347033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:33.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.537+0000 7f3e127fc640 1 -- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e1415d8d0 msgr2=0x7f3e140cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.537+0000 7f3e127fc640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3e1415d8d0 0x7f3e140cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.537+0000 7f3e127fc640 1 -- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e14071210 msgr2=0x7f3e141f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.537+0000 7f3e127fc640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3e14071210 0x7f3e141f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.537+0000 7f3e127fc640 1 -- 172.21.15.129:0/2598347033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e04009400 con 0x7f3e14143680 2024-03-23T16:01:33.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.540+0000 7f3e127fc640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e14143680 0x7f3e140b29f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c390 tx=0x7f3e0800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:33.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.544+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08005ce0 con 0x7f3e14143680 2024-03-23T16:01:33.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.544+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e08005e60 con 0x7f3e14143680 2024-03-23T16:01:33.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.544+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08007310 con 0x7f3e14143680 2024-03-23T16:01:33.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.544+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e140ce480 con 0x7f3e14143680 2024-03-23T16:01:33.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.544+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e140cbe50 con 0x7f3e14143680 2024-03-23T16:01:33.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.545+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e1415e380 con 0x7f3e14143680 2024-03-23T16:01:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.575+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e08007470 con 0x7f3e14143680 2024-03-23T16:01:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.576+0000 7f3dfbfff640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0006b860 0x7f3e0006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.576+0000 7f3e12ffd640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0006b860 0x7f3e0006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.577+0000 7f3e12ffd640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0006b860 0x7f3e0006dd20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3e04036000 tx=0x7f3e04008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:33.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.584+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3e08007720 con 0x7f3e14143680 2024-03-23T16:01:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.584+0000 7f3dfbfff640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f3e00073b60 0x7f3e00075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.584+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 --> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e00076650 con 0x7f3e00073b60 2024-03-23T16:01:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.584+0000 7f3e137fe640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f3e00073b60 0x7f3e00075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.584+0000 7f3e137fe640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f3e00073b60 0x7f3e00075f80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:33.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.585+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== osd.1 v2:172.21.15.129:6816/168045261 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e00076650 con 0x7f3e00073b60 2024-03-23T16:01:33.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.587+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e08007ea0 con 0x7f3e14143680 2024-03-23T16:01:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.599+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 --> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3e1415e0a0 con 0x7f3e00073b60 2024-03-23T16:01:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.599+0000 7f3dfbfff640 1 -- 172.21.15.129:0/2598347033 <== osd.1 v2:172.21.15.129:6816/168045261 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3e0c00f040 con 0x7f3e00073b60 2024-03-23T16:01:33.601 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:33.601 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:33.601 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.599+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f3e00073b60 msgr2=0x7f3e00075f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.599+0000 7f3e199e4640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:6816/168045261,v1:172.21.15.129:6817/168045261] conn(0x7f3e00073b60 0x7f3e00075f80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0006b860 msgr2=0x7f3e0006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3e0006b860 0x7f3e0006dd20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3e04036000 tx=0x7f3e04008030 comp rx=0 tx=0).stop 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e14143680 msgr2=0x7f3e140b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 --2- 172.21.15.129:0/2598347033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e14143680 0x7f3e140b29f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c390 tx=0x7f3e0800c850 comp rx=0 tx=0).stop 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e137fe640 1 -- 172.21.15.129:0/2598347033 reap_dead start 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 shutdown_connections 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 >> 172.21.15.129:0/2598347033 conn(0x7f3e1406b310 msgr2=0x7f3e1406c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 shutdown_connections 2024-03-23T16:01:33.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.600+0000 7f3e199e4640 1 -- 172.21.15.129:0/2598347033 wait complete. 2024-03-23T16:01:33.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-23T16:01:33.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.713+0000 7fbb1dbaf640 1 Processor -- start 2024-03-23T16:01:33.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.713+0000 7fbb1dbaf640 1 -- start start 2024-03-23T16:01:33.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.714+0000 7fbb1dbaf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbb18143710 0x7fbb1806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.714+0000 7fbb1dbaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 0x7fbb180cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.714+0000 7fbb1dbaf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1806d140 con 0x7fbb18070840 2024-03-23T16:01:33.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.714+0000 7fbb1dbaf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1815db10 con 0x7fbb18143710 2024-03-23T16:01:33.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.714+0000 7fbb1dbaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1815ddf0 con 0x7fbb18070420 2024-03-23T16:01:33.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.715+0000 7fbb177fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbb18143710 0x7fbb1806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.715+0000 7fbb16ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 0x7fbb180cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.715+0000 7fbb16ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 0x7fbb180cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:39640/0 (socket says 172.21.15.129:39640) 2024-03-23T16:01:33.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.715+0000 7fbb16ffd640 1 -- 172.21.15.129:0/2884599166 learned_addr learned my addr 172.21.15.129:0/2884599166 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:33.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.718+0000 7fbb16ffd640 1 -- 172.21.15.129:0/2884599166 >> v2:172.21.15.157:3300/0 conn(0x7fbb18143710 msgr2=0x7fbb1806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:33.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.718+0000 7fbb16ffd640 1 --2- 172.21.15.129:0/2884599166 >> v2:172.21.15.157:3300/0 conn(0x7fbb18143710 0x7fbb1806cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.718+0000 7fbb16ffd640 1 -- 172.21.15.129:0/2884599166 >> v1:172.21.15.157:6789/0 conn(0x7fbb18070840 legacy=0x7fbb180c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.718+0000 7fbb16ffd640 1 -- 172.21.15.129:0/2884599166 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1815e0d0 con 0x7fbb18070420 2024-03-23T16:01:33.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.721+0000 7fbb16ffd640 1 --2- 172.21.15.129:0/2884599166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 0x7fbb180cd880 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009870 tx=0x7fbb0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a62619e5865a637 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:33.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.727+0000 7fbb167fc640 1 -- 172.21.15.129:0/2884599166 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c030840 con 0x7fbb18070420 2024-03-23T16:01:33.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.727+0000 7fbb167fc640 1 -- 172.21.15.129:0/2884599166 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb0c0309a0 con 0x7fbb18070420 2024-03-23T16:01:33.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.727+0000 7fbb167fc640 1 -- 172.21.15.129:0/2884599166 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c030d30 con 0x7fbb18070420 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/2884599166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 msgr2=0x7fbb180cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/2884599166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 0x7fbb180cd880 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009870 tx=0x7fbb0c02b980 comp rx=0 tx=0).stop 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/2884599166 shutdown_connections 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/2884599166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070420 0x7fbb180cd880 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/2884599166 >> v2:172.21.15.157:3300/0 conn(0x7fbb18143710 0x7fbb1806cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/2884599166 >> 172.21.15.129:0/2884599166 conn(0x7fbb18061b50 msgr2=0x7fbb18061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/2884599166 shutdown_connections 2024-03-23T16:01:33.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.728+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/2884599166 wait complete. 2024-03-23T16:01:33.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.729+0000 7fbb1dbaf640 1 Processor -- start 2024-03-23T16:01:33.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.729+0000 7fbb1dbaf640 1 -- start start 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.730+0000 7fbb1dbaf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb18070420 0x7fbb180b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb1dbaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070840 0x7fbb18154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb1dbaf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb18143710 0x7fbb18155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb1dbaf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181f0c80 con 0x7fbb18070420 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb1dbaf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbb181f0320 con 0x7fbb18143710 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb1dbaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb181f4330 con 0x7fbb18070840 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb17fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb18143710 0x7fbb18155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.731+0000 7fbb16ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070840 0x7fbb18154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.732+0000 7fbb16ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070840 0x7fbb18154c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:39648/0 (socket says 172.21.15.129:39648) 2024-03-23T16:01:33.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.732+0000 7fbb16ffd640 1 -- 172.21.15.129:0/3688826075 learned_addr learned my addr 172.21.15.129:0/3688826075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:33.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.733+0000 7fbb177fe640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb18070420 0x7fbb180b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:33.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.735+0000 7fbb17fff640 1 -- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070840 msgr2=0x7fbb18154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.735+0000 7fbb17fff640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070840 0x7fbb18154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.735+0000 7fbb17fff640 1 -- 172.21.15.129:0/3688826075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb18070420 msgr2=0x7fbb180b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:33.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.735+0000 7fbb17fff640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb18070420 0x7fbb180b22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:33.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.735+0000 7fbb17fff640 1 -- 172.21.15.129:0/3688826075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c0094d0 con 0x7fbb18143710 2024-03-23T16:01:33.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.738+0000 7fbb17fff640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb18143710 0x7fbb18155760 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbb180ce9f0 tx=0x7fbb1000cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.744+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10003b70 con 0x7fbb18143710 2024-03-23T16:01:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.744+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb10003cd0 con 0x7fbb18143710 2024-03-23T16:01:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.744+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10005310 con 0x7fbb18143710 2024-03-23T16:01:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.744+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb181f05c0 con 0x7fbb18143710 2024-03-23T16:01:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.744+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb18157370 con 0x7fbb18143710 2024-03-23T16:01:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:33.745+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb181f0dc0 con 0x7fbb18143710 2024-03-23T16:01:34.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.015+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb10005470 con 0x7fbb18143710 2024-03-23T16:01:34.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.016+0000 7fbafffff640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbb0406b860 0x7fbb0406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.016+0000 7fbb177fe640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbb0406b860 0x7fbb0406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.016+0000 7fbb177fe640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbb0406b860 0x7fbb0406dd20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009aa0 tx=0x7fbb08009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.023+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fbb10058f00 con 0x7fbb18143710 2024-03-23T16:01:34.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.041+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb100551a0 con 0x7fbb18143710 2024-03-23T16:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.223+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fbb180cdfd0 con 0x7fbb0406b860 2024-03-23T16:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.224+0000 7fbafffff640 1 -- 172.21.15.129:0/3688826075 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbb180cdfd0 con 0x7fbb0406b860 2024-03-23T16:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 1 to deep-scrub 2024-03-23T16:01:34.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbb0406b860 msgr2=0x7fbb0406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbb0406b860 0x7fbb0406dd20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009aa0 tx=0x7fbb08009190 comp rx=0 tx=0).stop 2024-03-23T16:01:34.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb18143710 msgr2=0x7fbb18155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb18143710 0x7fbb18155760 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbb180ce9f0 tx=0x7fbb1000cb20 comp rx=0 tx=0).stop 2024-03-23T16:01:34.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 shutdown_connections 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbb0406b860 0x7fbb0406dd20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb18143710 0x7fbb18155760 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb18070840 0x7fbb18154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 --2- 172.21.15.129:0/3688826075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb18070420 0x7fbb180b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 >> 172.21.15.129:0/3688826075 conn(0x7fbb18061b50 msgr2=0x7fbb18140e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 shutdown_connections 2024-03-23T16:01:34.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.226+0000 7fbb1dbaf640 1 -- 172.21.15.129:0/3688826075 wait complete. 2024-03-23T16:01:34.240 INFO:tasks.ceph:Scrubbing osd.2 2024-03-23T16:01:34.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:34.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.347+0000 7f1c974df640 1 Processor -- start 2024-03-23T16:01:34.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.347+0000 7f1c974df640 1 -- start start 2024-03-23T16:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.349+0000 7f1c974df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c9006c970 0x7f1c900cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.349+0000 7f1c974df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1c9015dc10 0x7f1c9015b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.349+0000 7f1c974df640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c900cf820 con 0x7f1c9015d830 2024-03-23T16:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.349+0000 7f1c974df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1c900ce680 con 0x7f1c9015dc10 2024-03-23T16:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.349+0000 7f1c974df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9015e0a0 con 0x7f1c9006c970 2024-03-23T16:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.349+0000 7f1c95a7c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1c9015dc10 0x7f1c9015b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.350+0000 7f1c94a7a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c9006c970 0x7f1c900cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.350+0000 7f1c95a7c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1c9015dc10 0x7f1c9015b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44940/0 (socket says 172.21.15.129:44940) 2024-03-23T16:01:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.350+0000 7f1c95a7c640 1 -- 172.21.15.129:0/487734722 learned_addr learned my addr 172.21.15.129:0/487734722 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.353+0000 7f1c94a7a640 1 -- 172.21.15.129:0/487734722 >> v2:172.21.15.157:3300/0 conn(0x7f1c9015dc10 msgr2=0x7f1c9015b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.353+0000 7f1c94a7a640 1 --2- 172.21.15.129:0/487734722 >> v2:172.21.15.157:3300/0 conn(0x7f1c9015dc10 0x7f1c9015b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.353+0000 7f1c94a7a640 1 -- 172.21.15.129:0/487734722 >> v1:172.21.15.157:6789/0 conn(0x7f1c9015d830 legacy=0x7f1c9006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.353+0000 7f1c94a7a640 1 -- 172.21.15.129:0/487734722 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c900cda70 con 0x7f1c9006c970 2024-03-23T16:01:34.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.356+0000 7f1c94a7a640 1 --2- 172.21.15.129:0/487734722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c9006c970 0x7f1c900cf1e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c800097a0 tx=0x7f1c8002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b54542c6d7574d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.364+0000 7f1c87fff640 1 -- 172.21.15.129:0/487734722 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8002e070 con 0x7f1c9006c970 2024-03-23T16:01:34.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.364+0000 7f1c87fff640 1 -- 172.21.15.129:0/487734722 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c800315a0 con 0x7f1c9006c970 2024-03-23T16:01:34.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.364+0000 7f1c87fff640 1 -- 172.21.15.129:0/487734722 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80031930 con 0x7f1c9006c970 2024-03-23T16:01:34.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 -- 172.21.15.129:0/487734722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c9006c970 msgr2=0x7f1c900cf1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 --2- 172.21.15.129:0/487734722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c9006c970 0x7f1c900cf1e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c800097a0 tx=0x7f1c8002b840 comp rx=0 tx=0).stop 2024-03-23T16:01:34.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 -- 172.21.15.129:0/487734722 shutdown_connections 2024-03-23T16:01:34.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 --2- 172.21.15.129:0/487734722 >> v2:172.21.15.157:3300/0 conn(0x7f1c9015dc10 0x7f1c9015b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 --2- 172.21.15.129:0/487734722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c9006c970 0x7f1c900cf1e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 -- 172.21.15.129:0/487734722 >> 172.21.15.129:0/487734722 conn(0x7f1c900b1650 msgr2=0x7f1c900b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:34.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 -- 172.21.15.129:0/487734722 shutdown_connections 2024-03-23T16:01:34.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.365+0000 7f1c974df640 1 -- 172.21.15.129:0/487734722 wait complete. 2024-03-23T16:01:34.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.367+0000 7f1c974df640 1 Processor -- start 2024-03-23T16:01:34.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.367+0000 7f1c974df640 1 -- start start 2024-03-23T16:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c974df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c9015d830 0x7f1c9015a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c974df640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c9015dc10 0x7f1c90154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c974df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c90154110 0x7f1c900b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c974df640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9015af20 con 0x7f1c9015dc10 2024-03-23T16:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c974df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1c901f0dc0 con 0x7f1c9015d830 2024-03-23T16:01:34.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c974df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1c90156990 con 0x7f1c90154110 2024-03-23T16:01:34.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c94a7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c9015dc10 0x7f1c90154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c95a7c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c90154110 0x7f1c900b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.368+0000 7f1c9527b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c9015d830 0x7f1c9015a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.369+0000 7f1c94a7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c9015dc10 0x7f1c90154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33710/0 (socket says 172.21.15.129:33710) 2024-03-23T16:01:34.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.369+0000 7f1c94a7a640 1 -- 172.21.15.129:0/2639006212 learned_addr learned my addr 172.21.15.129:0/2639006212 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.372+0000 7f1c94a7a640 1 -- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c90154110 msgr2=0x7f1c900b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.372+0000 7f1c94a7a640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1c90154110 0x7f1c900b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.372+0000 7f1c94a7a640 1 -- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c9015d830 msgr2=0x7f1c9015a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.372+0000 7f1c94a7a640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c9015d830 0x7f1c9015a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.372+0000 7f1c94a7a640 1 -- 172.21.15.129:0/2639006212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c80009400 con 0x7f1c9015dc10 2024-03-23T16:01:34.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.374+0000 7f1c94a7a640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c9015dc10 0x7f1c90154b30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1c800098c0 tx=0x7f1c800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.378+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80032840 con 0x7f1c9015dc10 2024-03-23T16:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.378+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c800329c0 con 0x7f1c9015dc10 2024-03-23T16:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.378+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c901efef0 con 0x7f1c9015dc10 2024-03-23T16:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.378+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80032d70 con 0x7f1c9015dc10 2024-03-23T16:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.378+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c901f0490 con 0x7f1c9015dc10 2024-03-23T16:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.379+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c900d09f0 con 0x7f1c9015dc10 2024-03-23T16:01:34.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.399+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1c800453f0 con 0x7f1c9015dc10 2024-03-23T16:01:34.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.399+0000 7f1c85ffb640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1c7406b840 0x7f1c7406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.400+0000 7f1c9527b640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1c7406b840 0x7f1c7406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.400+0000 7f1c9527b640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1c7406b840 0x7f1c7406dd00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c006fd0 tx=0x7f1c8c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.403+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f1c8007b280 con 0x7f1c9015dc10 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.403+0000 7f1c85ffb640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f1c74073b50 0x7f1c74075f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.403+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 --> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c74076640 con 0x7f1c74073b50 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.403+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1c80077560 con 0x7f1c9015dc10 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.403+0000 7f1c95a7c640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f1c74073b50 0x7f1c74075f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.403+0000 7f1c95a7c640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f1c74073b50 0x7f1c74075f70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.404+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== osd.2 v2:172.21.15.129:6800/2679851475 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c74076640 con 0x7f1c74073b50 2024-03-23T16:01:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 --> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1c90156b50 con 0x7f1c74073b50 2024-03-23T16:01:34.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c85ffb640 1 -- 172.21.15.129:0/2639006212 <== osd.2 v2:172.21.15.129:6800/2679851475 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1c90156b50 con 0x7f1c74073b50 2024-03-23T16:01:34.420 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:34.420 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:34.420 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f1c74073b50 msgr2=0x7f1c74075f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:6800/2679851475,v1:172.21.15.129:6801/2679851475] conn(0x7f1c74073b50 0x7f1c74075f70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1c7406b840 msgr2=0x7f1c7406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f1c7406b840 0x7f1c7406dd00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c006fd0 tx=0x7f1c8c009060 comp rx=0 tx=0).stop 2024-03-23T16:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c9015dc10 msgr2=0x7f1c90154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c974df640 1 --2- 172.21.15.129:0/2639006212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c9015dc10 0x7f1c90154b30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1c800098c0 tx=0x7f1c800046c0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.417+0000 7f1c95a7c640 1 -- 172.21.15.129:0/2639006212 reap_dead start 2024-03-23T16:01:34.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.418+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 shutdown_connections 2024-03-23T16:01:34.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.418+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 >> 172.21.15.129:0/2639006212 conn(0x7f1c900b1650 msgr2=0x7f1c900c8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:34.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.418+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 shutdown_connections 2024-03-23T16:01:34.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.418+0000 7f1c974df640 1 -- 172.21.15.129:0/2639006212 wait complete. 2024-03-23T16:01:34.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-23T16:01:34.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.532+0000 7fbe80a62640 1 Processor -- start 2024-03-23T16:01:34.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.532+0000 7fbe80a62640 1 -- start start 2024-03-23T16:01:34.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.533+0000 7fbe80a62640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbe7c0cd9d0 0x7fbe7c0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.533+0000 7fbe80a62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c0c98e0 0x7fbe7c158100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.533+0000 7fbe80a62640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe7c158d00 con 0x7fbe7c1523c0 2024-03-23T16:01:34.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.533+0000 7fbe80a62640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbe7c0c9cf0 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.533+0000 7fbe80a62640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7c0c9f80 con 0x7fbe7c0c98e0 2024-03-23T16:01:34.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.534+0000 7fbe7a59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fbe7c1523c0 0x7fbe7c0cad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55170/0 (socket says 172.21.15.129:55170) 2024-03-23T16:01:34.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.534+0000 7fbe7a59c640 1 -- 172.21.15.129:0/2562579647 learned_addr learned my addr 172.21.15.129:0/2562579647 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:34.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.534+0000 7fbe7ad9d640 1 --2- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c0c98e0 0x7fbe7c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.534+0000 7fbe79d9b640 1 --2- 172.21.15.129:0/2562579647 >> v2:172.21.15.157:3300/0 conn(0x7fbe7c0cd9d0 0x7fbe7c0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.536+0000 7fbe7959a640 1 -- 172.21.15.129:0/2562579647 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569773576 0 0) 0x7fbe7c158d00 con 0x7fbe7c1523c0 2024-03-23T16:01:34.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.536+0000 7fbe7959a640 1 -- 172.21.15.129:0/2562579647 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe64003520 con 0x7fbe7c1523c0 2024-03-23T16:01:34.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.537+0000 7fbe79d9b640 1 -- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c0c98e0 msgr2=0x7fbe7c158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:34.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.537+0000 7fbe79d9b640 1 --2- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c0c98e0 0x7fbe7c158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.537+0000 7fbe79d9b640 1 -- 172.21.15.129:0/2562579647 >> v1:172.21.15.129:6789/0 conn(0x7fbe7c1523c0 legacy=0x7fbe7c0cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.537+0000 7fbe79d9b640 1 -- 172.21.15.129:0/2562579647 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7c0ca260 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.539+0000 7fbe79d9b640 1 --2- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c93a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe7002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ca65f8fcb2afb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.543+0000 7fbe7959a640 1 -- 172.21.15.129:0/2562579647 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70031830 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.543+0000 7fbe7959a640 1 -- 172.21.15.129:0/2562579647 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe70031990 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.543+0000 7fbe7959a640 1 -- 172.21.15.129:0/2562579647 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70031d20 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 -- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0cd9d0 msgr2=0x7fbe7c0c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 --2- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c93a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe7002b8d0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 -- 172.21.15.129:0/2562579647 shutdown_connections 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 --2- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c0c98e0 0x7fbe7c158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 --2- 172.21.15.129:0/2562579647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c93a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 -- 172.21.15.129:0/2562579647 >> 172.21.15.129:0/2562579647 conn(0x7fbe7c06f7d0 msgr2=0x7fbe7c06fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 -- 172.21.15.129:0/2562579647 shutdown_connections 2024-03-23T16:01:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.544+0000 7fbe80a62640 1 -- 172.21.15.129:0/2562579647 wait complete. 2024-03-23T16:01:34.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.545+0000 7fbe80a62640 1 Processor -- start 2024-03-23T16:01:34.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.545+0000 7fbe80a62640 1 -- start start 2024-03-23T16:01:34.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.546+0000 7fbe80a62640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0c98e0 0x7fbe7c1f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.546+0000 7fbe80a62640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.546+0000 7fbe80a62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c1523c0 0x7fbe7c0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.546+0000 7fbe80a62640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7c157df0 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.546+0000 7fbe80a62640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7c156e00 con 0x7fbe7c0c98e0 2024-03-23T16:01:34.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.546+0000 7fbe80a62640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7c0c7cb0 con 0x7fbe7c1523c0 2024-03-23T16:01:34.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.547+0000 7fbe79d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.547+0000 7fbe7a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0c98e0 0x7fbe7c1f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.547+0000 7fbe7ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c1523c0 0x7fbe7c0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.547+0000 7fbe79d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33718/0 (socket says 172.21.15.129:33718) 2024-03-23T16:01:34.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.547+0000 7fbe79d9b640 1 -- 172.21.15.129:0/3698182611 learned_addr learned my addr 172.21.15.129:0/3698182611 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:34.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.550+0000 7fbe79d9b640 1 -- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c1523c0 msgr2=0x7fbe7c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.550+0000 7fbe79d9b640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c1523c0 0x7fbe7c0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.550+0000 7fbe79d9b640 1 -- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0c98e0 msgr2=0x7fbe7c1f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:34.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.550+0000 7fbe79d9b640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0c98e0 0x7fbe7c1f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:34.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.550+0000 7fbe79d9b640 1 -- 172.21.15.129:0/3698182611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe70009400 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.553+0000 7fbe79d9b640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c83a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbe7002bee0 tx=0x7fbe700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.557+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7003a630 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.557+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7003a7b0 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.557+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7c1f1dd0 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.557+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7003ab60 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.557+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7c158020 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.558+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe7c0757b0 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.815+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbe7003acc0 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.815+0000 7fbe6affd640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe5c06b810 0x7fbe5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:34.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.816+0000 7fbe7a59c640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe5c06b810 0x7fbe5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:34.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.816+0000 7fbe7a59c640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe5c06b810 0x7fbe5c06dcd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c001200 tx=0x7fbe6c000bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:34.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.829+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fbe7007a960 con 0x7fbe7c0cd9d0 2024-03-23T16:01:34.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:34.829+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbe70076c00 con 0x7fbe7c0cd9d0 2024-03-23T16:01:35.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.040+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fbe7c1527a0 con 0x7fbe5c06b810 2024-03-23T16:01:35.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.041+0000 7fbe6affd640 1 -- 172.21.15.129:0/3698182611 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbe6c0014f0 con 0x7fbe5c06b810 2024-03-23T16:01:35.043 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 2 to deep-scrub 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe5c06b810 msgr2=0x7fbe5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe5c06b810 0x7fbe5c06dcd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c001200 tx=0x7fbe6c000bc0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 msgr2=0x7fbe7c0c83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c83a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbe7002bee0 tx=0x7fbe700046c0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 shutdown_connections 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe7c1523c0 0x7fbe7c0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe7c0cd9d0 0x7fbe7c0c83a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fbe5c06b810 0x7fbe5c06dcd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 --2- 172.21.15.129:0/3698182611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe7c0c98e0 0x7fbe7c1f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 >> 172.21.15.129:0/3698182611 conn(0x7fbe7c06f7d0 msgr2=0x7fbe7c0cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:35.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 shutdown_connections 2024-03-23T16:01:35.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.043+0000 7fbe80a62640 1 -- 172.21.15.129:0/3698182611 wait complete. 2024-03-23T16:01:35.056 INFO:tasks.ceph:Scrubbing osd.3 2024-03-23T16:01:35.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:35.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.160+0000 7f0c60193640 1 Processor -- start 2024-03-23T16:01:35.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.161+0000 7f0c60193640 1 -- start start 2024-03-23T16:01:35.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.161+0000 7f0c60193640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5806dbb0 0x7f0c580cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.161+0000 7f0c60193640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5815a420 con 0x7f0c5806bb70 2024-03-23T16:01:35.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.161+0000 7f0c60193640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5806df90 con 0x7f0c5815b130 2024-03-23T16:01:35.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.161+0000 7f0c60193640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0c580ccae0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.162+0000 7f0c5d72e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0c5815b130 0x7f0c580cf4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43690/0 (socket says 172.21.15.129:43690) 2024-03-23T16:01:35.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.162+0000 7f0c5d72e640 1 -- 172.21.15.129:0/3094164183 learned_addr learned my addr 172.21.15.129:0/3094164183 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:35.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.162+0000 7f0c5df2f640 1 --2- 172.21.15.129:0/3094164183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5806dbb0 0x7f0c580cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.165+0000 7f0c5cf2d640 1 -- 172.21.15.129:0/3094164183 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120075785 0 0) 0x7f0c5815a420 con 0x7f0c5806bb70 2024-03-23T16:01:35.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.165+0000 7f0c5cf2d640 1 -- 172.21.15.129:0/3094164183 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c44003520 con 0x7f0c5806bb70 2024-03-23T16:01:35.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.166+0000 7f0c5df2f640 1 -- 172.21.15.129:0/3094164183 >> v1:172.21.15.157:6789/0 conn(0x7f0c5815b130 legacy=0x7f0c580cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.166+0000 7f0c5df2f640 1 -- 172.21.15.129:0/3094164183 >> v1:172.21.15.129:6789/0 conn(0x7f0c5806bb70 legacy=0x7f0c580ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.166+0000 7f0c5df2f640 1 -- 172.21.15.129:0/3094164183 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c580ccdc0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.168+0000 7f0c5df2f640 1 --2- 172.21.15.129:0/3094164183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5806dbb0 0x7f0c580cc3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009800 tx=0x7f0c5402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a542a27cde81b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.172+0000 7f0c5cf2d640 1 -- 172.21.15.129:0/3094164183 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54030880 con 0x7f0c5806dbb0 2024-03-23T16:01:35.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.172+0000 7f0c5cf2d640 1 -- 172.21.15.129:0/3094164183 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c540309e0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.172+0000 7f0c5cf2d640 1 -- 172.21.15.129:0/3094164183 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54030d70 con 0x7f0c5806dbb0 2024-03-23T16:01:35.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 -- 172.21.15.129:0/3094164183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5806dbb0 msgr2=0x7f0c580cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 --2- 172.21.15.129:0/3094164183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5806dbb0 0x7f0c580cc3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009800 tx=0x7f0c5402b9c0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 -- 172.21.15.129:0/3094164183 shutdown_connections 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 --2- 172.21.15.129:0/3094164183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5806dbb0 0x7f0c580cc3f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 -- 172.21.15.129:0/3094164183 >> 172.21.15.129:0/3094164183 conn(0x7f0c580cfbb0 msgr2=0x7f0c580cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 -- 172.21.15.129:0/3094164183 shutdown_connections 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.173+0000 7f0c60193640 1 -- 172.21.15.129:0/3094164183 wait complete. 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.174+0000 7f0c60193640 1 Processor -- start 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.174+0000 7f0c60193640 1 -- start start 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.175+0000 7f0c60193640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5806bb70 0x7f0c5814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.175+0000 7f0c60193640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c5806dbb0 0x7f0c581f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.175+0000 7f0c60193640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5815b130 0x7f0c581f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.175+0000 7f0c60193640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0c580b3360 con 0x7f0c5806dbb0 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.175+0000 7f0c60193640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0c580b3640 con 0x7f0c5806bb70 2024-03-23T16:01:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.175+0000 7f0c60193640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0c580b3920 con 0x7f0c5815b130 2024-03-23T16:01:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.176+0000 7f0c5d72e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c5806dbb0 0x7f0c581f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.176+0000 7f0c5e730640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5815b130 0x7f0c581f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.176+0000 7f0c5df2f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5806bb70 0x7f0c5814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.176+0000 7f0c5d72e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c5806dbb0 0x7f0c581f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33724/0 (socket says 172.21.15.129:33724) 2024-03-23T16:01:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.176+0000 7f0c5d72e640 1 -- 172.21.15.129:0/3001377834 learned_addr learned my addr 172.21.15.129:0/3001377834 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.180+0000 7f0c5d72e640 1 -- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5815b130 msgr2=0x7f0c581f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.180+0000 7f0c5d72e640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5815b130 0x7f0c581f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.180+0000 7f0c5d72e640 1 -- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5806bb70 msgr2=0x7f0c5814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.180+0000 7f0c5d72e640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5806bb70 0x7f0c5814e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.180+0000 7f0c5d72e640 1 -- 172.21.15.129:0/3001377834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c54009400 con 0x7f0c5806dbb0 2024-03-23T16:01:35.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.182+0000 7f0c5d72e640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c5806dbb0 0x7f0c581f1910 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800c3c0 tx=0x7f0c4800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.186+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48005ce0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.187+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c48005e40 con 0x7f0c5806dbb0 2024-03-23T16:01:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.187+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c580b3c00 con 0x7f0c5806dbb0 2024-03-23T16:01:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.187+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480072d0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.187+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c580b41c0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.188+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c5815ac10 con 0x7f0c5806dbb0 2024-03-23T16:01:35.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.208+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c480074f0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.209+0000 7f0c4e7fc640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0c3c06b860 0x7f0c3c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.209+0000 7f0c5df2f640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0c3c06b860 0x7f0c3c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.209+0000 7f0c5df2f640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0c3c06b860 0x7f0c3c06dd20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0c54036000 tx=0x7f0c54008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.213+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0c48058620 con 0x7f0c5806dbb0 2024-03-23T16:01:35.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.213+0000 7f0c4e7fc640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f0c3c073b60 0x7f0c3c075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.213+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 --> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c3c076650 con 0x7f0c3c073b60 2024-03-23T16:01:35.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.213+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0c480548c0 con 0x7f0c5806dbb0 2024-03-23T16:01:35.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.214+0000 7f0c5e730640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f0c3c073b60 0x7f0c3c075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.214+0000 7f0c5e730640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f0c3c073b60 0x7f0c3c075f80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.215+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== osd.3 v2:172.21.15.129:6802/3775172849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c3c076650 con 0x7f0c3c073b60 2024-03-23T16:01:35.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 --> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0c5815a930 con 0x7f0c3c073b60 2024-03-23T16:01:35.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c4e7fc640 1 -- 172.21.15.129:0/3001377834 <== osd.3 v2:172.21.15.129:6802/3775172849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0c50001ac0 con 0x7f0c3c073b60 2024-03-23T16:01:35.232 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:35.232 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:35.232 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:35.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f0c3c073b60 msgr2=0x7f0c3c075f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:6802/3775172849,v1:172.21.15.129:6804/3775172849] conn(0x7f0c3c073b60 0x7f0c3c075f80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0c3c06b860 msgr2=0x7f0c3c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0c3c06b860 0x7f0c3c06dd20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0c54036000 tx=0x7f0c54008030 comp rx=0 tx=0).stop 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c5806dbb0 msgr2=0x7f0c581f1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 --2- 172.21.15.129:0/3001377834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c5806dbb0 0x7f0c581f1910 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800c3c0 tx=0x7f0c4800c880 comp rx=0 tx=0).stop 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c5e730640 1 -- 172.21.15.129:0/3001377834 reap_dead start 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 shutdown_connections 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 >> 172.21.15.129:0/3001377834 conn(0x7f0c580cfbb0 msgr2=0x7f0c58159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.230+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 shutdown_connections 2024-03-23T16:01:35.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.231+0000 7f0c60193640 1 -- 172.21.15.129:0/3001377834 wait complete. 2024-03-23T16:01:35.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-23T16:01:35.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.347+0000 7f7b73e8f640 1 Processor -- start 2024-03-23T16:01:35.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.347+0000 7f7b73e8f640 1 -- start start 2024-03-23T16:01:35.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.348+0000 7f7b73e8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c143710 0x7f7b6c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.348+0000 7f7b73e8f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7b6c070840 0x7f7b6c0c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.348+0000 7f7b73e8f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b6c06d140 con 0x7f7b6c070420 2024-03-23T16:01:35.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.348+0000 7f7b73e8f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7b6c15db10 con 0x7f7b6c070840 2024-03-23T16:01:35.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.348+0000 7f7b73e8f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6c15ddf0 con 0x7f7b6c143710 2024-03-23T16:01:35.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.349+0000 7f7b7142a640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7b6c070420 0x7f7b6c0cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55188/0 (socket says 172.21.15.129:55188) 2024-03-23T16:01:35.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.349+0000 7f7b7142a640 1 -- 172.21.15.129:0/1915361292 learned_addr learned my addr 172.21.15.129:0/1915361292 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:35.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.349+0000 7f7b71c2b640 1 --2- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c143710 0x7f7b6c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.349+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1915361292 >> v2:172.21.15.157:3300/0 conn(0x7f7b6c070840 0x7f7b6c0c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.352+0000 7f7b70c29640 1 -- 172.21.15.129:0/1915361292 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662630774 0 0) 0x7f7b6c06d140 con 0x7f7b6c070420 2024-03-23T16:01:35.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.352+0000 7f7b70c29640 1 -- 172.21.15.129:0/1915361292 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b58003520 con 0x7f7b6c070420 2024-03-23T16:01:35.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.352+0000 7f7b7242c640 1 -- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c143710 msgr2=0x7f7b6c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:35.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.353+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c143710 0x7f7b6c06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.353+0000 7f7b7242c640 1 -- 172.21.15.129:0/1915361292 >> v1:172.21.15.129:6789/0 conn(0x7f7b6c070420 legacy=0x7f7b6c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.353+0000 7f7b7242c640 1 -- 172.21.15.129:0/1915361292 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6c15e0d0 con 0x7f7b6c070840 2024-03-23T16:01:35.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.355+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c0c2f00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009d90 tx=0x7f7b6402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4587d2f59a37e962 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b70c29640 1 -- 172.21.15.129:0/1915361292 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6402cc60 con 0x7f7b6c070840 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b70c29640 1 -- 172.21.15.129:0/1915361292 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b6402cdc0 con 0x7f7b6c070840 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b70c29640 1 -- 172.21.15.129:0/1915361292 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64034660 con 0x7f7b6c070840 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 msgr2=0x7f7b6c0c2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c0c2f00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009d90 tx=0x7f7b6402beb0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1915361292 shutdown_connections 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c0c2f00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.361+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1915361292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c143710 0x7f7b6c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.362+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1915361292 >> 172.21.15.129:0/1915361292 conn(0x7f7b6c061b50 msgr2=0x7f7b6c061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.362+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1915361292 shutdown_connections 2024-03-23T16:01:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.362+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1915361292 wait complete. 2024-03-23T16:01:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 Processor -- start 2024-03-23T16:01:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 -- start start 2024-03-23T16:01:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c070420 0x7f7b6c1ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6c143710 0x7f7b6c140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c1568d0 con 0x7f7b6c143710 2024-03-23T16:01:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6c155a50 con 0x7f7b6c070840 2024-03-23T16:01:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.363+0000 7f7b73e8f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6c157810 con 0x7f7b6c070420 2024-03-23T16:01:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.364+0000 7f7b71c2b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c070420 0x7f7b6c1ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.364+0000 7f7b7142a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.364+0000 7f7b7142a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c140140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44988/0 (socket says 172.21.15.129:44988) 2024-03-23T16:01:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.364+0000 7f7b7142a640 1 -- 172.21.15.129:0/1396604135 learned_addr learned my addr 172.21.15.129:0/1396604135 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:35.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.365+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6c143710 0x7f7b6c140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.369+0000 7f7b7242c640 1 -- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c070420 msgr2=0x7f7b6c1ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.369+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c070420 0x7f7b6c1ef9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.369+0000 7f7b7242c640 1 -- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 msgr2=0x7f7b6c140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.369+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c140140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.369+0000 7f7b7242c640 1 -- 172.21.15.129:0/1396604135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b64009a70 con 0x7f7b6c143710 2024-03-23T16:01:35.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.371+0000 7f7b7242c640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6c143710 0x7f7b6c140680 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7b64033000 tx=0x7f7b64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.375+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64040650 con 0x7f7b6c143710 2024-03-23T16:01:35.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.375+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b640407b0 con 0x7f7b6c143710 2024-03-23T16:01:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.376+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6c157d60 con 0x7f7b6c143710 2024-03-23T16:01:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.376+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64040b60 con 0x7f7b6c143710 2024-03-23T16:01:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.376+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b6c155400 con 0x7f7b6c143710 2024-03-23T16:01:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.377+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b6c155bc0 con 0x7f7b6c143710 2024-03-23T16:01:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.395+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7b64040cc0 con 0x7f7b6c143710 2024-03-23T16:01:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.395+0000 7f7b627fc640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b5006b860 0x7f7b5006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.396+0000 7f7b71c2b640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b5006b860 0x7f7b5006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.396+0000 7f7b71c2b640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b5006b860 0x7f7b5006dd20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009aa0 tx=0x7f7b68009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.399+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f7b6407b910 con 0x7f7b6c143710 2024-03-23T16:01:35.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.644+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7b64077bb0 con 0x7f7b6c143710 2024-03-23T16:01:35.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.843+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f7b6c0c8790 con 0x7f7b5006b860 2024-03-23T16:01:35.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.844+0000 7f7b627fc640 1 -- 172.21.15.129:0/1396604135 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7b6c0c8790 con 0x7f7b5006b860 2024-03-23T16:01:35.846 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 3 to deep-scrub 2024-03-23T16:01:35.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b5006b860 msgr2=0x7f7b5006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b5006b860 0x7f7b5006dd20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009aa0 tx=0x7f7b68009190 comp rx=0 tx=0).stop 2024-03-23T16:01:35.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6c143710 msgr2=0x7f7b6c140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6c143710 0x7f7b6c140680 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7b64033000 tx=0x7f7b64004a40 comp rx=0 tx=0).stop 2024-03-23T16:01:35.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 shutdown_connections 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b6c143710 0x7f7b6c140680 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f7b5006b860 0x7f7b5006dd20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b6c070840 0x7f7b6c140140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 --2- 172.21.15.129:0/1396604135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b6c070420 0x7f7b6c1ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 >> 172.21.15.129:0/1396604135 conn(0x7f7b6c061b50 msgr2=0x7f7b6c142200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 shutdown_connections 2024-03-23T16:01:35.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.847+0000 7f7b73e8f640 1 -- 172.21.15.129:0/1396604135 wait complete. 2024-03-23T16:01:35.864 INFO:tasks.ceph:Scrubbing osd.4 2024-03-23T16:01:35.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:35.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.972+0000 7f3aab264640 1 Processor -- start 2024-03-23T16:01:35.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.972+0000 7f3aab264640 1 -- start start 2024-03-23T16:01:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.973+0000 7f3aab264640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3aa406c2a0 0x7f3aa4165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.973+0000 7f3aab264640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa40ce530 con 0x7f3aa415db40 2024-03-23T16:01:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.973+0000 7f3aab264640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa40ce9c0 con 0x7f3aa415d760 2024-03-23T16:01:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.973+0000 7f3aab264640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa415e4e0 con 0x7f3aa406c2a0 2024-03-23T16:01:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.974+0000 7f3aa9801640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3aa406c2a0 0x7f3aa4165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.974+0000 7f3aa9000640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3aa415d760 0x7f3aa415b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43706/0 (socket says 172.21.15.129:43706) 2024-03-23T16:01:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.974+0000 7f3aa9000640 1 -- 172.21.15.129:0/1938943715 learned_addr learned my addr 172.21.15.129:0/1938943715 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:35.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.977+0000 7f3aa9801640 1 -- 172.21.15.129:0/1938943715 >> v1:172.21.15.157:6789/0 conn(0x7f3aa415d760 legacy=0x7f3aa415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.977+0000 7f3aa9801640 1 -- 172.21.15.129:0/1938943715 >> v1:172.21.15.129:6789/0 conn(0x7f3aa415db40 legacy=0x7f3aa406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.977+0000 7f3aa9801640 1 -- 172.21.15.129:0/1938943715 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa406ca40 con 0x7f3aa406c2a0 2024-03-23T16:01:35.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.981+0000 7f3aa9801640 1 --2- 172.21.15.129:0/1938943715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa406c2a0 0x7f3aa4165b30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009d90 tx=0x7f3a9c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22750434d6fabb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:35.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.987+0000 7f3aa37fe640 1 -- 172.21.15.129:0/1938943715 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c02cd00 con 0x7f3aa406c2a0 2024-03-23T16:01:35.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.987+0000 7f3aa37fe640 1 -- 172.21.15.129:0/1938943715 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c02ce60 con 0x7f3aa406c2a0 2024-03-23T16:01:35.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.987+0000 7f3aa37fe640 1 -- 172.21.15.129:0/1938943715 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c034660 con 0x7f3aa406c2a0 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 -- 172.21.15.129:0/1938943715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa406c2a0 msgr2=0x7f3aa4165b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 --2- 172.21.15.129:0/1938943715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa406c2a0 0x7f3aa4165b30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009d90 tx=0x7f3a9c02bf50 comp rx=0 tx=0).stop 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 -- 172.21.15.129:0/1938943715 shutdown_connections 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 --2- 172.21.15.129:0/1938943715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa406c2a0 0x7f3aa4165b30 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 -- 172.21.15.129:0/1938943715 >> 172.21.15.129:0/1938943715 conn(0x7f3aa40b1630 msgr2=0x7f3aa40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 -- 172.21.15.129:0/1938943715 shutdown_connections 2024-03-23T16:01:35.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.988+0000 7f3aab264640 1 -- 172.21.15.129:0/1938943715 wait complete. 2024-03-23T16:01:35.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.989+0000 7f3aab264640 1 Processor -- start 2024-03-23T16:01:35.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.989+0000 7f3aab264640 1 -- start start 2024-03-23T16:01:35.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aab264640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3aa406c2a0 0x7f3aa40b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aab264640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3aa415d760 0x7f3aa40cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aab264640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa415db40 0x7f3aa41e3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:35.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aab264640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa40c8b60 con 0x7f3aa406c2a0 2024-03-23T16:01:35.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aab264640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa40b3a30 con 0x7f3aa415db40 2024-03-23T16:01:35.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aab264640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa41e5e40 con 0x7f3aa415d760 2024-03-23T16:01:35.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aa9801640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa415db40 0x7f3aa41e3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.990+0000 7f3aa9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3aa406c2a0 0x7f3aa40b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.991+0000 7f3aa9801640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa415db40 0x7f3aa41e3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:44992/0 (socket says 172.21.15.129:44992) 2024-03-23T16:01:35.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.991+0000 7f3aa9801640 1 -- 172.21.15.129:0/384294644 learned_addr learned my addr 172.21.15.129:0/384294644 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:35.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.991+0000 7f3aa3fff640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3aa415d760 0x7f3aa40cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:35.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.994+0000 7f3aa9801640 1 -- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3aa415d760 msgr2=0x7f3aa40cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.994+0000 7f3aa9801640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3aa415d760 0x7f3aa40cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.994+0000 7f3aa9801640 1 -- 172.21.15.129:0/384294644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3aa406c2a0 msgr2=0x7f3aa40b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:35.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.994+0000 7f3aa9801640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3aa406c2a0 0x7f3aa40b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:35.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.994+0000 7f3aa9801640 1 -- 172.21.15.129:0/384294644 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c009a70 con 0x7f3aa415db40 2024-03-23T16:01:35.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.997+0000 7f3aa9801640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa415db40 0x7f3aa41e3900 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02c560 tx=0x7f3a9c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.999+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c03e610 con 0x7f3aa415db40 2024-03-23T16:01:36.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.999+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa41e5530 con 0x7f3aa415db40 2024-03-23T16:01:36.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:35.999+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa40c9070 con 0x7f3aa415db40 2024-03-23T16:01:36.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.000+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3aa40cdee0 con 0x7f3aa415db40 2024-03-23T16:01:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.001+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c03e770 con 0x7f3aa415db40 2024-03-23T16:01:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.001+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c03eae0 con 0x7f3aa415db40 2024-03-23T16:01:36.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.025+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3a9c03b050 con 0x7f3aa415db40 2024-03-23T16:01:36.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.025+0000 7f3aa17fa640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a8806b810 0x7f3a8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.025+0000 7f3aa9000640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a8806b810 0x7f3a8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.026+0000 7f3aa9000640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a8806b810 0x7f3a8806dcd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3a98007fd0 tx=0x7f3a98007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.030+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f3a9c07b7f0 con 0x7f3aa415db40 2024-03-23T16:01:36.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.031+0000 7f3aa17fa640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7f3a88073b10 0x7f3a88075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.031+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 --> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a88076600 con 0x7f3a88073b10 2024-03-23T16:01:36.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.031+0000 7f3aa3fff640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7f3a88073b10 0x7f3a88075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.031+0000 7f3aa3fff640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7f3a88073b10 0x7f3a88075f30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.031+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a9c077a90 con 0x7f3aa415db40 2024-03-23T16:01:36.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.032+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== osd.4 v2:172.21.15.157:6826/4230040520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3a88076600 con 0x7f3a88073b10 2024-03-23T16:01:36.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.045+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 --> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3aa40cdc00 con 0x7f3a88073b10 2024-03-23T16:01:36.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.045+0000 7f3aa17fa640 1 -- 172.21.15.129:0/384294644 <== osd.4 v2:172.21.15.157:6826/4230040520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3a90001810 con 0x7f3a88073b10 2024-03-23T16:01:36.047 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:36.047 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:36.048 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:36.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7f3a88073b10 msgr2=0x7f3a88075f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6826/4230040520,v1:172.21.15.157:6827/4230040520] conn(0x7f3a88073b10 0x7f3a88075f30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a8806b810 msgr2=0x7f3a8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f3a8806b810 0x7f3a8806dcd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3a98007fd0 tx=0x7f3a98007e80 comp rx=0 tx=0).stop 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa415db40 msgr2=0x7f3aa41e3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 --2- 172.21.15.129:0/384294644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3aa415db40 0x7f3aa41e3900 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02c560 tx=0x7f3a9c004a40 comp rx=0 tx=0).stop 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aa9801640 1 -- 172.21.15.129:0/384294644 reap_dead start 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 shutdown_connections 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 >> 172.21.15.129:0/384294644 conn(0x7f3aa40b1630 msgr2=0x7f3aa40b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 shutdown_connections 2024-03-23T16:01:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.046+0000 7f3aab264640 1 -- 172.21.15.129:0/384294644 wait complete. 2024-03-23T16:01:36.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-23T16:01:36.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.160+0000 7f04800eb640 1 Processor -- start 2024-03-23T16:01:36.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.160+0000 7f04800eb640 1 -- start start 2024-03-23T16:01:36.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.161+0000 7f04800eb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04780c7700 0x7f047815d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.161+0000 7f04800eb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04780cc2a0 con 0x7f0478154590 2024-03-23T16:01:36.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.161+0000 7f04800eb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f047815ab20 con 0x7f04780cb780 2024-03-23T16:01:36.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.161+0000 7f04800eb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f04780cbce0 con 0x7f04780c7700 2024-03-23T16:01:36.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.162+0000 7f047de87640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f0478154590 0x7f04780ca990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55214/0 (socket says 172.21.15.129:55214) 2024-03-23T16:01:36.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.162+0000 7f047de87640 1 -- 172.21.15.129:0/303717715 learned_addr learned my addr 172.21.15.129:0/303717715 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:36.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.162+0000 7f047e688640 1 --2- 172.21.15.129:0/303717715 >> v2:172.21.15.157:3300/0 conn(0x7f04780c7700 0x7f047815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.165+0000 7f047ce85640 1 -- 172.21.15.129:0/303717715 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171012607 0 0) 0x7f04780cc2a0 con 0x7f0478154590 2024-03-23T16:01:36.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.165+0000 7f047ce85640 1 -- 172.21.15.129:0/303717715 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0464003520 con 0x7f0478154590 2024-03-23T16:01:36.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.167+0000 7f047e688640 1 -- 172.21.15.129:0/303717715 >> v1:172.21.15.157:6789/0 conn(0x7f04780cb780 legacy=0x7f04780c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.167+0000 7f047e688640 1 -- 172.21.15.129:0/303717715 >> v1:172.21.15.129:6789/0 conn(0x7f0478154590 legacy=0x7f04780ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.167+0000 7f047e688640 1 -- 172.21.15.129:0/303717715 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04780cbfc0 con 0x7f04780c7700 2024-03-23T16:01:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.176+0000 7f047e688640 1 --2- 172.21.15.129:0/303717715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780c7700 0x7f047815d920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0470009d90 tx=0x7f047002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2f813290d863935 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.181+0000 7f047ce85640 1 -- 172.21.15.129:0/303717715 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047002cc60 con 0x7f04780c7700 2024-03-23T16:01:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.181+0000 7f047ce85640 1 -- 172.21.15.129:0/303717715 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f047002cdc0 con 0x7f04780c7700 2024-03-23T16:01:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.181+0000 7f047ce85640 1 -- 172.21.15.129:0/303717715 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470034660 con 0x7f04780c7700 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 -- 172.21.15.129:0/303717715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780c7700 msgr2=0x7f047815d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 --2- 172.21.15.129:0/303717715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780c7700 0x7f047815d920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0470009d90 tx=0x7f047002beb0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 -- 172.21.15.129:0/303717715 shutdown_connections 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 --2- 172.21.15.129:0/303717715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780c7700 0x7f047815d920 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 -- 172.21.15.129:0/303717715 >> 172.21.15.129:0/303717715 conn(0x7f0478061b50 msgr2=0x7f0478061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 -- 172.21.15.129:0/303717715 shutdown_connections 2024-03-23T16:01:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.182+0000 7f04800eb640 1 -- 172.21.15.129:0/303717715 wait complete. 2024-03-23T16:01:36.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.183+0000 7f04800eb640 1 Processor -- start 2024-03-23T16:01:36.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.183+0000 7f04800eb640 1 -- start start 2024-03-23T16:01:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f04800eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04780c7700 0x7f04780ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f04800eb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780cb780 0x7f04781f0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f04800eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0478154590 0x7f04781f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f04800eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04781f3140 con 0x7f04780c7700 2024-03-23T16:01:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f04800eb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04781f33e0 con 0x7f04780cb780 2024-03-23T16:01:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f04800eb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04781f36c0 con 0x7f0478154590 2024-03-23T16:01:36.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.184+0000 7f047de87640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04780c7700 0x7f04780ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.185+0000 7f047d686640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780cb780 0x7f04781f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.185+0000 7f047d686640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780cb780 0x7f04781f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45008/0 (socket says 172.21.15.129:45008) 2024-03-23T16:01:36.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.185+0000 7f047d686640 1 -- 172.21.15.129:0/3456755085 learned_addr learned my addr 172.21.15.129:0/3456755085 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:36.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.185+0000 7f047e688640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0478154590 0x7f04781f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.188+0000 7f047de87640 1 -- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0478154590 msgr2=0x7f04781f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.188+0000 7f047de87640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0478154590 0x7f04781f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.188+0000 7f047de87640 1 -- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780cb780 msgr2=0x7f04781f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.188+0000 7f047de87640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780cb780 0x7f04781f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.188+0000 7f047de87640 1 -- 172.21.15.129:0/3456755085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0470009a70 con 0x7f04780c7700 2024-03-23T16:01:36.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.191+0000 7f047de87640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04780c7700 0x7f04780ce1e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f047400c3c0 tx=0x7f0474001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.195+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474005ce0 con 0x7f04780c7700 2024-03-23T16:01:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.195+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0474005e40 con 0x7f04780c7700 2024-03-23T16:01:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.195+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04781f39a0 con 0x7f04780c7700 2024-03-23T16:01:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.195+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04740072d0 con 0x7f04780c7700 2024-03-23T16:01:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.195+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04781efce0 con 0x7f04780c7700 2024-03-23T16:01:36.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.196+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04780757b0 con 0x7f04780c7700 2024-03-23T16:01:36.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.217+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f04740074f0 con 0x7f04780c7700 2024-03-23T16:01:36.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.218+0000 7f046e7fc640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045c06b860 0x7f045c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.218+0000 7f047d686640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045c06b860 0x7f045c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.218+0000 7f047d686640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045c06b860 0x7f045c06dd20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f04781e4330 tx=0x7f0468009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.221+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0474058410 con 0x7f04780c7700 2024-03-23T16:01:36.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.468+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f04740546b0 con 0x7f04780c7700 2024-03-23T16:01:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.668+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f047815a7b0 con 0x7f045c06b860 2024-03-23T16:01:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.668+0000 7f046e7fc640 1 -- 172.21.15.129:0/3456755085 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0468001500 con 0x7f045c06b860 2024-03-23T16:01:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 4 to deep-scrub 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.670+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045c06b860 msgr2=0x7f045c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.670+0000 7f04800eb640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045c06b860 0x7f045c06dd20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f04781e4330 tx=0x7f0468009040 comp rx=0 tx=0).stop 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04780c7700 msgr2=0x7f04780ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04780c7700 0x7f04780ce1e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f047400c3c0 tx=0x7f0474001e70 comp rx=0 tx=0).stop 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 shutdown_connections 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0478154590 0x7f04781f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f045c06b860 0x7f045c06dd20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04780cb780 0x7f04781f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 --2- 172.21.15.129:0/3456755085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04780c7700 0x7f04780ce1e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 >> 172.21.15.129:0/3456755085 conn(0x7f0478061b50 msgr2=0x7f047814eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 shutdown_connections 2024-03-23T16:01:36.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.671+0000 7f04800eb640 1 -- 172.21.15.129:0/3456755085 wait complete. 2024-03-23T16:01:36.683 INFO:tasks.ceph:Scrubbing osd.5 2024-03-23T16:01:36.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.788+0000 7fac19aed640 1 Processor -- start 2024-03-23T16:01:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.788+0000 7fac19aed640 1 -- start start 2024-03-23T16:01:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac19aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac140cc4e0 0x7fac140cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac19aed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1406cb90 con 0x7fac140cb380 2024-03-23T16:01:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac19aed640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac140cb990 con 0x7fac1415a340 2024-03-23T16:01:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac19aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fac14154a70 con 0x7fac140cc4e0 2024-03-23T16:01:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac12ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fac140cb380 0x7fac1415d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:55218/0 (socket says 172.21.15.129:55218) 2024-03-23T16:01:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac13fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fac1415a340 0x7fac1415df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43732/0 (socket says 172.21.15.129:43732) 2024-03-23T16:01:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.789+0000 7fac12ffd640 1 -- 172.21.15.129:0/994823307 learned_addr learned my addr 172.21.15.129:0/994823307 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.790+0000 7fac137fe640 1 --2- 172.21.15.129:0/994823307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac140cc4e0 0x7fac140cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.793+0000 7fac127fc640 1 -- 172.21.15.129:0/994823307 <== mon.0 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 708542553 0 0) 0x7fac1406cb90 con 0x7fac140cb380 2024-03-23T16:01:36.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.793+0000 7fac127fc640 1 -- 172.21.15.129:0/994823307 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac00003520 con 0x7fac140cb380 2024-03-23T16:01:36.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.793+0000 7fac137fe640 1 -- 172.21.15.129:0/994823307 >> v1:172.21.15.157:6789/0 conn(0x7fac1415a340 legacy=0x7fac1415df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.793+0000 7fac137fe640 1 -- 172.21.15.129:0/994823307 >> v1:172.21.15.129:6789/0 conn(0x7fac140cb380 legacy=0x7fac1415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.793+0000 7fac137fe640 1 -- 172.21.15.129:0/994823307 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1415e660 con 0x7fac140cc4e0 2024-03-23T16:01:36.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.795+0000 7fac137fe640 1 --2- 172.21.15.129:0/994823307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac140cc4e0 0x7fac140cad50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac04009dc0 tx=0x7fac0402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c554c9776d9dbdf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.799+0000 7fac127fc640 1 -- 172.21.15.129:0/994823307 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0402cc40 con 0x7fac140cc4e0 2024-03-23T16:01:36.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.799+0000 7fac127fc640 1 -- 172.21.15.129:0/994823307 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0402cda0 con 0x7fac140cc4e0 2024-03-23T16:01:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.799+0000 7fac127fc640 1 -- 172.21.15.129:0/994823307 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04034660 con 0x7fac140cc4e0 2024-03-23T16:01:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 -- 172.21.15.129:0/994823307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac140cc4e0 msgr2=0x7fac140cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 --2- 172.21.15.129:0/994823307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac140cc4e0 0x7fac140cad50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac04009dc0 tx=0x7fac0402be90 comp rx=0 tx=0).stop 2024-03-23T16:01:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 -- 172.21.15.129:0/994823307 shutdown_connections 2024-03-23T16:01:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 --2- 172.21.15.129:0/994823307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac140cc4e0 0x7fac140cad50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 -- 172.21.15.129:0/994823307 >> 172.21.15.129:0/994823307 conn(0x7fac14062180 msgr2=0x7fac140b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 -- 172.21.15.129:0/994823307 shutdown_connections 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.800+0000 7fac19aed640 1 -- 172.21.15.129:0/994823307 wait complete. 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.801+0000 7fac19aed640 1 Processor -- start 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.801+0000 7fac19aed640 1 -- start start 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.802+0000 7fac19aed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac140cb380 0x7fac140c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.802+0000 7fac19aed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac140cc4e0 0x7fac141f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.802+0000 7fac19aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1415a340 0x7fac1414fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.802+0000 7fac19aed640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac1414f990 con 0x7fac140cc4e0 2024-03-23T16:01:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.802+0000 7fac19aed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac1414eab0 con 0x7fac140cb380 2024-03-23T16:01:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.802+0000 7fac19aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fac140c4510 con 0x7fac1415a340 2024-03-23T16:01:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.803+0000 7fac13fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1415a340 0x7fac1414fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.803+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac140cc4e0 0x7fac141f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.803+0000 7fac137fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac140cb380 0x7fac140c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.803+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac140cc4e0 0x7fac141f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33748/0 (socket says 172.21.15.129:33748) 2024-03-23T16:01:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.803+0000 7fac12ffd640 1 -- 172.21.15.129:0/43843633 learned_addr learned my addr 172.21.15.129:0/43843633 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:36.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.807+0000 7fac12ffd640 1 -- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1415a340 msgr2=0x7fac1414fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.807+0000 7fac12ffd640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1415a340 0x7fac1414fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.807+0000 7fac12ffd640 1 -- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac140cb380 msgr2=0x7fac140c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.807+0000 7fac12ffd640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac140cb380 0x7fac140c3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.807+0000 7fac12ffd640 1 -- 172.21.15.129:0/43843633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac04009a70 con 0x7fac140cc4e0 2024-03-23T16:01:36.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.809+0000 7fac12ffd640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac140cc4e0 0x7fac141f3b70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00c410 tx=0x7fabfc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.813+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0074e0 con 0x7fac140cc4e0 2024-03-23T16:01:36.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.813+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabfc007640 con 0x7fac140cc4e0 2024-03-23T16:01:36.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.813+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc004a00 con 0x7fac140cc4e0 2024-03-23T16:01:36.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.813+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac14151c50 con 0x7fac140cc4e0 2024-03-23T16:01:36.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.813+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac14151970 con 0x7fac140cc4e0 2024-03-23T16:01:36.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.814+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac14159150 con 0x7fac140cc4e0 2024-03-23T16:01:36.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.844+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fabfc004b60 con 0x7fac140cc4e0 2024-03-23T16:01:36.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.844+0000 7fac0bfff640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabf406b890 0x7fabf406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.845+0000 7fac137fe640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabf406b890 0x7fabf406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.845+0000 7fac137fe640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabf406b890 0x7fabf406dd50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fac04036000 tx=0x7fac04030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.850+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fabfc058ee0 con 0x7fac140cc4e0 2024-03-23T16:01:36.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.850+0000 7fac0bfff640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fabf4073ba0 0x7fabf4075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.850+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 --> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabf4076690 con 0x7fabf4073ba0 2024-03-23T16:01:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.850+0000 7fac13fff640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fabf4073ba0 0x7fabf4075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.850+0000 7fac13fff640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fabf4073ba0 0x7fabf4075fc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.851+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== osd.5 v2:172.21.15.157:6802/869595834 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fabf4076690 con 0x7fabf4073ba0 2024-03-23T16:01:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.851+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fabfc055180 con 0x7fac140cc4e0 2024-03-23T16:01:36.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.878+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 --> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fac1406c5d0 con 0x7fabf4073ba0 2024-03-23T16:01:36.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.878+0000 7fac0bfff640 1 -- 172.21.15.129:0/43843633 <== osd.5 v2:172.21.15.157:6802/869595834 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fac0c007300 con 0x7fabf4073ba0 2024-03-23T16:01:36.881 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:36.881 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:36.881 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fabf4073ba0 msgr2=0x7fabf4075fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6802/869595834,v1:172.21.15.157:6803/869595834] conn(0x7fabf4073ba0 0x7fabf4075fc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabf406b890 msgr2=0x7fabf406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fabf406b890 0x7fabf406dd50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fac04036000 tx=0x7fac04030000 comp rx=0 tx=0).stop 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac140cc4e0 msgr2=0x7fac141f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 --2- 172.21.15.129:0/43843633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac140cc4e0 0x7fac141f3b70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00c410 tx=0x7fabfc00c8d0 comp rx=0 tx=0).stop 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac13fff640 1 -- 172.21.15.129:0/43843633 reap_dead start 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 shutdown_connections 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 >> 172.21.15.129:0/43843633 conn(0x7fac14062180 msgr2=0x7fac140afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 shutdown_connections 2024-03-23T16:01:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.879+0000 7fac19aed640 1 -- 172.21.15.129:0/43843633 wait complete. 2024-03-23T16:01:36.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-23T16:01:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.996+0000 7fb5b968d640 1 Processor -- start 2024-03-23T16:01:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.996+0000 7fb5b968d640 1 -- start start 2024-03-23T16:01:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.997+0000 7fb5b968d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b4154590 0x7fb5b415d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.997+0000 7fb5b968d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb5b415afb0 0x7fb5b40c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.997+0000 7fb5b968d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b40cc2a0 con 0x7fb5b40cb780 2024-03-23T16:01:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.997+0000 7fb5b968d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb5b415ab20 con 0x7fb5b415afb0 2024-03-23T16:01:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.997+0000 7fb5b968d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b40cbce0 con 0x7fb5b4154590 2024-03-23T16:01:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.998+0000 7fb5b37fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb5b415afb0 0x7fb5b40c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.998+0000 7fb5abfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb5b40cb780 0x7fb5b40ca990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43744/0 (socket says 172.21.15.129:43744) 2024-03-23T16:01:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:36.998+0000 7fb5abfff640 1 -- 172.21.15.129:0/2972024192 learned_addr learned my addr 172.21.15.129:0/2972024192 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:37.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.002+0000 7fb5b37fe640 1 -- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b4154590 msgr2=0x7fb5b415d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.002+0000 7fb5b37fe640 1 --2- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b4154590 0x7fb5b415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.002+0000 7fb5b37fe640 1 -- 172.21.15.129:0/2972024192 >> v1:172.21.15.157:6789/0 conn(0x7fb5b40cb780 legacy=0x7fb5b40ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.002+0000 7fb5b37fe640 1 -- 172.21.15.129:0/2972024192 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b40cbfc0 con 0x7fb5b415afb0 2024-03-23T16:01:37.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.006+0000 7fb5b37fe640 1 --2- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b415afb0 0x7fb5b40c6ff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009d90 tx=0x7fb5ac02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=436006339cef7118 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.012+0000 7fb5b27fc640 1 -- 172.21.15.129:0/2972024192 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac02cbc0 con 0x7fb5b415afb0 2024-03-23T16:01:37.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.012+0000 7fb5b27fc640 1 -- 172.21.15.129:0/2972024192 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ac02cd20 con 0x7fb5b415afb0 2024-03-23T16:01:37.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.012+0000 7fb5b27fc640 1 -- 172.21.15.129:0/2972024192 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac034660 con 0x7fb5b415afb0 2024-03-23T16:01:37.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 -- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b415afb0 msgr2=0x7fb5b40c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 --2- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b415afb0 0x7fb5b40c6ff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009d90 tx=0x7fb5ac02be10 comp rx=0 tx=0).stop 2024-03-23T16:01:37.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 -- 172.21.15.129:0/2972024192 shutdown_connections 2024-03-23T16:01:37.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 --2- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b415afb0 0x7fb5b40c6ff0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 --2- 172.21.15.129:0/2972024192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b4154590 0x7fb5b415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 -- 172.21.15.129:0/2972024192 >> 172.21.15.129:0/2972024192 conn(0x7fb5b4061b50 msgr2=0x7fb5b4061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:37.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.013+0000 7fb5b968d640 1 -- 172.21.15.129:0/2972024192 shutdown_connections 2024-03-23T16:01:37.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.014+0000 7fb5b968d640 1 -- 172.21.15.129:0/2972024192 wait complete. 2024-03-23T16:01:37.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.015+0000 7fb5b968d640 1 Processor -- start 2024-03-23T16:01:37.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.015+0000 7fb5b968d640 1 -- start start 2024-03-23T16:01:37.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b968d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5b40cb780 0x7fb5b4155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b968d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b4154590 0x7fb5b4150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b968d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b415afb0 0x7fb5b4147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b968d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b40cd9f0 con 0x7fb5b40cb780 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b968d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b40cdc40 con 0x7fb5b4154590 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b968d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b41572d0 con 0x7fb5b415afb0 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b4154590 0x7fb5b4150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.017+0000 7fb5b37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b415afb0 0x7fb5b4147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.018+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b4154590 0x7fb5b4150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45038/0 (socket says 172.21.15.129:45038) 2024-03-23T16:01:37.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.018+0000 7fb5abfff640 1 -- 172.21.15.129:0/1934404590 learned_addr learned my addr 172.21.15.129:0/1934404590 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:37.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.022+0000 7fb5b37fe640 1 -- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b4154590 msgr2=0x7fb5b4150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.022+0000 7fb5b37fe640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b4154590 0x7fb5b4150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.022+0000 7fb5b37fe640 1 -- 172.21.15.129:0/1934404590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5b40cb780 msgr2=0x7fb5b4155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.022+0000 7fb5b37fe640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5b40cb780 0x7fb5b4155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.022+0000 7fb5b37fe640 1 -- 172.21.15.129:0/1934404590 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac009a70 con 0x7fb5b415afb0 2024-03-23T16:01:37.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.024+0000 7fb5b37fe640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b415afb0 0x7fb5b4147d10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac033000 tx=0x7fb5ac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.027+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac040650 con 0x7fb5b415afb0 2024-03-23T16:01:37.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.027+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b41575b0 con 0x7fb5b415afb0 2024-03-23T16:01:37.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.027+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b41559d0 con 0x7fb5b415afb0 2024-03-23T16:01:37.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.028+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b4155c10 con 0x7fb5b415afb0 2024-03-23T16:01:37.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.305+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ac02ce90 con 0x7fb5b415afb0 2024-03-23T16:01:37.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.305+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0409b0 con 0x7fb5b415afb0 2024-03-23T16:01:37.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.305+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb5ac040bd0 con 0x7fb5b415afb0 2024-03-23T16:01:37.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.306+0000 7fb5ab7fe640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb59406b840 0x7fb59406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.306+0000 7fb5b2ffd640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb59406b840 0x7fb59406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.306+0000 7fb5b2ffd640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb59406b840 0x7fb59406dd00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009710 tx=0x7fb5a4006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.311+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb5ac07c250 con 0x7fb5b415afb0 2024-03-23T16:01:37.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.326+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb5ac077df0 con 0x7fb5b415afb0 2024-03-23T16:01:37.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.517+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fb5b4154970 con 0x7fb59406b840 2024-03-23T16:01:37.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.518+0000 7fb5ab7fe640 1 -- 172.21.15.129:0/1934404590 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb5a400b480 con 0x7fb59406b840 2024-03-23T16:01:37.521 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 5 to deep-scrub 2024-03-23T16:01:37.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb59406b840 msgr2=0x7fb59406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb59406b840 0x7fb59406dd00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009710 tx=0x7fb5a4006c90 comp rx=0 tx=0).stop 2024-03-23T16:01:37.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b415afb0 msgr2=0x7fb5b4147d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b415afb0 0x7fb5b4147d10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac033000 tx=0x7fb5ac004a40 comp rx=0 tx=0).stop 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 shutdown_connections 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5b415afb0 0x7fb5b4147d10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb59406b840 0x7fb59406dd00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5b4154590 0x7fb5b4150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 --2- 172.21.15.129:0/1934404590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5b40cb780 0x7fb5b4155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 >> 172.21.15.129:0/1934404590 conn(0x7fb5b4061b50 msgr2=0x7fb5b4159b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 shutdown_connections 2024-03-23T16:01:37.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.520+0000 7fb5b968d640 1 -- 172.21.15.129:0/1934404590 wait complete. 2024-03-23T16:01:37.533 INFO:tasks.ceph:Scrubbing osd.6 2024-03-23T16:01:37.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.637+0000 7f2d0bdc7640 1 Processor -- start 2024-03-23T16:01:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.637+0000 7f2d0bdc7640 1 -- start start 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.638+0000 7f2d0bdc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d040c8190 0x7f2d040c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.638+0000 7f2d0bdc7640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d04157180 con 0x7f2d04150fe0 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.638+0000 7f2d0bdc7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d04157d20 con 0x7f2d040c5950 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.638+0000 7f2d0bdc7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d040725f0 con 0x7f2d040c8190 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.639+0000 7f2d09362640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2d040c5950 0x7f2d040729d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43754/0 (socket says 172.21.15.129:43754) 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.639+0000 7f2d09362640 1 -- 172.21.15.129:0/1603947690 learned_addr learned my addr 172.21.15.129:0/1603947690 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.639+0000 7f2d0a364640 1 --2- 172.21.15.129:0/1603947690 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d040c8190 0x7f2d040c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.642+0000 7f2d08b61640 1 -- 172.21.15.129:0/1603947690 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713051535 0 0) 0x7f2d04157d20 con 0x7f2d040c5950 2024-03-23T16:01:37.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.642+0000 7f2d08b61640 1 -- 172.21.15.129:0/1603947690 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cec003520 con 0x7f2d040c5950 2024-03-23T16:01:37.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.643+0000 7f2d0a364640 1 -- 172.21.15.129:0/1603947690 >> v1:172.21.15.157:6789/0 conn(0x7f2d040c5950 legacy=0x7f2d040729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.643+0000 7f2d0a364640 1 -- 172.21.15.129:0/1603947690 >> v1:172.21.15.129:6789/0 conn(0x7f2d04150fe0 legacy=0x7f2d04157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.643+0000 7f2d0a364640 1 -- 172.21.15.129:0/1603947690 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d040730e0 con 0x7f2d040c8190 2024-03-23T16:01:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.645+0000 7f2d0a364640 1 --2- 172.21.15.129:0/1603947690 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d040c8190 0x7f2d040c85e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0097d0 tx=0x7f2cfc02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba7793d71754b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d08b61640 1 -- 172.21.15.129:0/1603947690 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc030880 con 0x7f2d040c8190 2024-03-23T16:01:37.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d08b61640 1 -- 172.21.15.129:0/1603947690 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cfc0309e0 con 0x7f2d040c8190 2024-03-23T16:01:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d08b61640 1 -- 172.21.15.129:0/1603947690 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc030d70 con 0x7f2d040c8190 2024-03-23T16:01:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/1603947690 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d040c8190 msgr2=0x7f2d040c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d0bdc7640 1 --2- 172.21.15.129:0/1603947690 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d040c8190 0x7f2d040c85e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0097d0 tx=0x7f2cfc02b9c0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/1603947690 shutdown_connections 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d0bdc7640 1 --2- 172.21.15.129:0/1603947690 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d040c8190 0x7f2d040c85e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.650+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/1603947690 >> 172.21.15.129:0/1603947690 conn(0x7f2d0406bc40 msgr2=0x7f2d0406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.651+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/1603947690 shutdown_connections 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.651+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/1603947690 wait complete. 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 Processor -- start 2024-03-23T16:01:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 -- start start 2024-03-23T16:01:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d040c5950 0x7f2d04156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d040c8190 0x7f2d0414d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d04150fe0 0x7f2d04142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0414cf50 con 0x7f2d040c5950 2024-03-23T16:01:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0414be30 con 0x7f2d040c8190 2024-03-23T16:01:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.652+0000 7f2d0bdc7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0414c110 con 0x7f2d04150fe0 2024-03-23T16:01:37.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.653+0000 7f2d09362640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d040c8190 0x7f2d0414d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.653+0000 7f2d0a364640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d04150fe0 0x7f2d04142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.653+0000 7f2d09b63640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d040c5950 0x7f2d04156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.653+0000 7f2d09362640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d040c8190 0x7f2d0414d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45040/0 (socket says 172.21.15.129:45040) 2024-03-23T16:01:37.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.653+0000 7f2d09362640 1 -- 172.21.15.129:0/3763141773 learned_addr learned my addr 172.21.15.129:0/3763141773 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.657+0000 7f2d09362640 1 -- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d04150fe0 msgr2=0x7f2d04142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.657+0000 7f2d09362640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d04150fe0 0x7f2d04142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.657+0000 7f2d09362640 1 -- 172.21.15.129:0/3763141773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d040c5950 msgr2=0x7f2d04156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.657+0000 7f2d09362640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d040c5950 0x7f2d04156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.657+0000 7f2d09362640 1 -- 172.21.15.129:0/3763141773 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cfc009400 con 0x7f2d040c8190 2024-03-23T16:01:37.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.660+0000 7f2d09362640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d040c8190 0x7f2d0414d3e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800c500 tx=0x7f2cf800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.665+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8005ce0 con 0x7f2d040c8190 2024-03-23T16:01:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.665+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf8005e40 con 0x7f2d040c8190 2024-03-23T16:01:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.665+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8007330 con 0x7f2d040c8190 2024-03-23T16:01:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.665+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0414c3f0 con 0x7f2d040c8190 2024-03-23T16:01:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.665+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d04156560 con 0x7f2d040c8190 2024-03-23T16:01:37.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.666+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d040c3220 con 0x7f2d040c8190 2024-03-23T16:01:37.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.692+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2cf8007490 con 0x7f2d040c8190 2024-03-23T16:01:37.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.693+0000 7f2d027fc640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cf006b890 0x7f2cf006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.693+0000 7f2d09b63640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cf006b890 0x7f2cf006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.694+0000 7f2d09b63640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cf006b890 0x7f2cf006dd50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4001e70 tx=0x7f2cf4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.696+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f2cf8007740 con 0x7f2d040c8190 2024-03-23T16:01:37.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.696+0000 7f2d027fc640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f2cf0073b90 0x7f2cf0075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.696+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 --> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cf0076680 con 0x7f2cf0073b90 2024-03-23T16:01:37.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.696+0000 7f2d0a364640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f2cf0073b90 0x7f2cf0075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.697+0000 7f2d0a364640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f2cf0073b90 0x7f2cf0075fb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.698+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2cf8058550 con 0x7f2d040c8190 2024-03-23T16:01:37.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.698+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== osd.6 v2:172.21.15.157:6810/4270775080 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2cf0076680 con 0x7f2cf0073b90 2024-03-23T16:01:37.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.711+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 --> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2d0414c280 con 0x7f2cf0073b90 2024-03-23T16:01:37.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.711+0000 7f2d027fc640 1 -- 172.21.15.129:0/3763141773 <== osd.6 v2:172.21.15.157:6810/4270775080 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2d0414c280 con 0x7f2cf0073b90 2024-03-23T16:01:37.713 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:37.713 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:37.713 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.711+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f2cf0073b90 msgr2=0x7f2cf0075fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.711+0000 7f2d0bdc7640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6810/4270775080,v1:172.21.15.157:6811/4270775080] conn(0x7f2cf0073b90 0x7f2cf0075fb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cf006b890 msgr2=0x7f2cf006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f2cf006b890 0x7f2cf006dd50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4001e70 tx=0x7f2cf4009040 comp rx=0 tx=0).stop 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d040c8190 msgr2=0x7f2d0414d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 --2- 172.21.15.129:0/3763141773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d040c8190 0x7f2d0414d3e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800c500 tx=0x7f2cf800c9c0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0a364640 1 -- 172.21.15.129:0/3763141773 reap_dead start 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 shutdown_connections 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 >> 172.21.15.129:0/3763141773 conn(0x7f2d0406bc40 msgr2=0x7f2d040c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 shutdown_connections 2024-03-23T16:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.712+0000 7f2d0bdc7640 1 -- 172.21.15.129:0/3763141773 wait complete. 2024-03-23T16:01:37.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-23T16:01:37.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.826+0000 7fdfb1047640 1 Processor -- start 2024-03-23T16:01:37.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.826+0000 7fdfb1047640 1 -- start start 2024-03-23T16:01:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.827+0000 7fdfb1047640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdfac0ce990 0x7fdfac0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.827+0000 7fdfb1047640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdfac06d410 0x7fdfac06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.827+0000 7fdfb1047640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfac0cfa30 con 0x7fdfac15aea0 2024-03-23T16:01:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.827+0000 7fdfb1047640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fdfac154bc0 con 0x7fdfac0ce990 2024-03-23T16:01:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.827+0000 7fdfb1047640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdfac0cf0e0 con 0x7fdfac06d410 2024-03-23T16:01:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.827+0000 7fdfa3fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdfac0ce990 0x7fdfac0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.828+0000 7fdfa3fff640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fdfac0ce990 0x7fdfac0ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33776/0 (socket says 172.21.15.129:33776) 2024-03-23T16:01:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.828+0000 7fdfa3fff640 1 -- 172.21.15.129:0/3525935107 learned_addr learned my addr 172.21.15.129:0/3525935107 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.828+0000 7fdfab59e640 1 --2- 172.21.15.129:0/3525935107 >> v2:172.21.15.157:3300/0 conn(0x7fdfac06d410 0x7fdfac06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.830+0000 7fdfa3fff640 1 -- 172.21.15.129:0/3525935107 >> v2:172.21.15.157:3300/0 conn(0x7fdfac06d410 msgr2=0x7fdfac06d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.830+0000 7fdfa3fff640 1 --2- 172.21.15.129:0/3525935107 >> v2:172.21.15.157:3300/0 conn(0x7fdfac06d410 0x7fdfac06d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.830+0000 7fdfa3fff640 1 -- 172.21.15.129:0/3525935107 >> v1:172.21.15.157:6789/0 conn(0x7fdfac15aea0 legacy=0x7fdfac0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.830+0000 7fdfa3fff640 1 -- 172.21.15.129:0/3525935107 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac0cdb50 con 0x7fdfac0ce990 2024-03-23T16:01:37.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.832+0000 7fdfa3fff640 1 --2- 172.21.15.129:0/3525935107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac0ce990 0x7fdfac0ce390 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdfac06d370 tx=0x7fdf9402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a238252da2e867 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.835+0000 7fdfaa59c640 1 -- 172.21.15.129:0/3525935107 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94004230 con 0x7fdfac0ce990 2024-03-23T16:01:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.835+0000 7fdfaa59c640 1 -- 172.21.15.129:0/3525935107 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf94004390 con 0x7fdfac0ce990 2024-03-23T16:01:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.835+0000 7fdfaa59c640 1 -- 172.21.15.129:0/3525935107 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94003810 con 0x7fdfac0ce990 2024-03-23T16:01:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 -- 172.21.15.129:0/3525935107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac0ce990 msgr2=0x7fdfac0ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 --2- 172.21.15.129:0/3525935107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac0ce990 0x7fdfac0ce390 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdfac06d370 tx=0x7fdf9402b8f0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 -- 172.21.15.129:0/3525935107 shutdown_connections 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 --2- 172.21.15.129:0/3525935107 >> v2:172.21.15.157:3300/0 conn(0x7fdfac06d410 0x7fdfac06d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 --2- 172.21.15.129:0/3525935107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac0ce990 0x7fdfac0ce390 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 -- 172.21.15.129:0/3525935107 >> 172.21.15.129:0/3525935107 conn(0x7fdfac0cfc20 msgr2=0x7fdfac0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 -- 172.21.15.129:0/3525935107 shutdown_connections 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.836+0000 7fdfb1047640 1 -- 172.21.15.129:0/3525935107 wait complete. 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 Processor -- start 2024-03-23T16:01:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 -- start start 2024-03-23T16:01:37.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac06d410 0x7fdfac1f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfac0ce990 0x7fdfac1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac15aea0 0x7fdfac0ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac0b48f0 con 0x7fdfac06d410 2024-03-23T16:01:37.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac151b90 con 0x7fdfac15aea0 2024-03-23T16:01:37.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.838+0000 7fdfb1047640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdfac1f1a50 con 0x7fdfac0ce990 2024-03-23T16:01:37.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.839+0000 7fdfab59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac15aea0 0x7fdfac0ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.839+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfac0ce990 0x7fdfac1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.839+0000 7fdfaad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac06d410 0x7fdfac1f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.840+0000 7fdfab59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac15aea0 0x7fdfac0ae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45064/0 (socket says 172.21.15.129:45064) 2024-03-23T16:01:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.840+0000 7fdfab59e640 1 -- 172.21.15.129:0/2784637740 learned_addr learned my addr 172.21.15.129:0/2784637740 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.843+0000 7fdfaad9d640 1 -- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfac0ce990 msgr2=0x7fdfac1f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.843+0000 7fdfaad9d640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfac0ce990 0x7fdfac1f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.843+0000 7fdfaad9d640 1 -- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac15aea0 msgr2=0x7fdfac0ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.843+0000 7fdfaad9d640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac15aea0 0x7fdfac0ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.843+0000 7fdfaad9d640 1 -- 172.21.15.129:0/2784637740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf94009460 con 0x7fdfac06d410 2024-03-23T16:01:37.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.846+0000 7fdfaad9d640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac06d410 0x7fdfac1f1ee0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00c3f0 tx=0x7fdf9c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.850+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0052d0 con 0x7fdfac06d410 2024-03-23T16:01:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.850+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c005450 con 0x7fdfac06d410 2024-03-23T16:01:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.850+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac157bd0 con 0x7fdfac06d410 2024-03-23T16:01:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.850+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c007260 con 0x7fdfac06d410 2024-03-23T16:01:37.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.850+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac0b4330 con 0x7fdfac06d410 2024-03-23T16:01:37.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.851+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac0757b0 con 0x7fdfac06d410 2024-03-23T16:01:37.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.873+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdf9c0073c0 con 0x7fdfac06d410 2024-03-23T16:01:37.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.874+0000 7fdfa37fe640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf8c06b860 0x7fdf8c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:37.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.874+0000 7fdfa3fff640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf8c06b860 0x7fdf8c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:37.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.874+0000 7fdfa3fff640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf8c06b860 0x7fdf8c06dd20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdf94035000 tx=0x7fdf94008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:37.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:37.876+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fdf9c0584a0 con 0x7fdfac06d410 2024-03-23T16:01:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.140+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdf9c054740 con 0x7fdfac06d410 2024-03-23T16:01:38.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.360+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fdfac0ce0d0 con 0x7fdf8c06b860 2024-03-23T16:01:38.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.360+0000 7fdfa37fe640 1 -- 172.21.15.129:0/2784637740 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdf94005990 con 0x7fdf8c06b860 2024-03-23T16:01:38.362 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 6 to deep-scrub 2024-03-23T16:01:38.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.362+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf8c06b860 msgr2=0x7fdf8c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.362+0000 7fdfb1047640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf8c06b860 0x7fdf8c06dd20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdf94035000 tx=0x7fdf94008030 comp rx=0 tx=0).stop 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.362+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac06d410 msgr2=0x7fdfac1f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.362+0000 7fdfb1047640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac06d410 0x7fdfac1f1ee0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00c3f0 tx=0x7fdf9c00c8b0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.362+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 shutdown_connections 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fdf8c06b860 0x7fdf8c06dd20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfac15aea0 0x7fdfac0ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfac0ce990 0x7fdfac1f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 --2- 172.21.15.129:0/2784637740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfac06d410 0x7fdfac1f1ee0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 >> 172.21.15.129:0/2784637740 conn(0x7fdfac0cfc20 msgr2=0x7fdfac158ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 shutdown_connections 2024-03-23T16:01:38.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.363+0000 7fdfb1047640 1 -- 172.21.15.129:0/2784637740 wait complete. 2024-03-23T16:01:38.376 INFO:tasks.ceph:Scrubbing osd.7 2024-03-23T16:01:38.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-23T16:01:38.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.480+0000 7ff43c00e640 1 Processor -- start 2024-03-23T16:01:38.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.480+0000 7ff43c00e640 1 -- start start 2024-03-23T16:01:38.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.481+0000 7ff43c00e640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff43415db40 0x7ff43406bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.481+0000 7ff43c00e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4340ce530 con 0x7ff43415d760 2024-03-23T16:01:38.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.481+0000 7ff43c00e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4340ce9c0 con 0x7ff43406c0d0 2024-03-23T16:01:38.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.481+0000 7ff43c00e640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7ff43415e4e0 con 0x7ff43415db40 2024-03-23T16:01:38.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.482+0000 7ff4395a9640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7ff43415db40 0x7ff43406bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.482+0000 7ff43a5ab640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff43406c0d0 0x7ff434164b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.129:43772/0 (socket says 172.21.15.129:43772) 2024-03-23T16:01:38.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.482+0000 7ff43a5ab640 1 -- 172.21.15.129:0/2168987871 learned_addr learned my addr 172.21.15.129:0/2168987871 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-23T16:01:38.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.484+0000 7ff4395a9640 1 -- 172.21.15.129:0/2168987871 >> v1:172.21.15.157:6789/0 conn(0x7ff43406c0d0 legacy=0x7ff434164b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.484+0000 7ff4395a9640 1 -- 172.21.15.129:0/2168987871 >> v1:172.21.15.129:6789/0 conn(0x7ff43415d760 legacy=0x7ff43415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:38.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.485+0000 7ff4395a9640 1 -- 172.21.15.129:0/2168987871 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43406ca40 con 0x7ff43415db40 2024-03-23T16:01:38.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.487+0000 7ff4395a9640 1 --2- 172.21.15.129:0/2168987871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff43406bb90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff4240097a0 tx=0x7ff42402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3982505ececc96c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.491+0000 7ff438da8640 1 -- 172.21.15.129:0/2168987871 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424031830 con 0x7ff43415db40 2024-03-23T16:01:38.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.491+0000 7ff438da8640 1 -- 172.21.15.129:0/2168987871 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff424031990 con 0x7ff43415db40 2024-03-23T16:01:38.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.491+0000 7ff438da8640 1 -- 172.21.15.129:0/2168987871 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424031d20 con 0x7ff43415db40 2024-03-23T16:01:38.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 -- 172.21.15.129:0/2168987871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 msgr2=0x7ff43406bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 --2- 172.21.15.129:0/2168987871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff43406bb90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff4240097a0 tx=0x7ff42402b8d0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 -- 172.21.15.129:0/2168987871 shutdown_connections 2024-03-23T16:01:38.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 --2- 172.21.15.129:0/2168987871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff43406bb90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 -- 172.21.15.129:0/2168987871 >> 172.21.15.129:0/2168987871 conn(0x7ff4340b1630 msgr2=0x7ff4340b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:38.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 -- 172.21.15.129:0/2168987871 shutdown_connections 2024-03-23T16:01:38.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.492+0000 7ff43c00e640 1 -- 172.21.15.129:0/2168987871 wait complete. 2024-03-23T16:01:38.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.493+0000 7ff43c00e640 1 Processor -- start 2024-03-23T16:01:38.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 -- start start 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff43406c0d0 0x7ff4341f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff43415d760 0x7ff4340cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff4341e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4340b2fa0 con 0x7ff43415db40 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341f0ef0 con 0x7ff43406c0d0 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.494+0000 7ff43c00e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4340c6f50 con 0x7ff43415d760 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.495+0000 7ff439daa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff43406c0d0 0x7ff4341f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.495+0000 7ff43a5ab640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff4341e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.495+0000 7ff4395a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff43415d760 0x7ff4340cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.495+0000 7ff439daa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff43406c0d0 0x7ff4341f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45080/0 (socket says 172.21.15.129:45080) 2024-03-23T16:01:38.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.495+0000 7ff439daa640 1 -- 172.21.15.129:0/3897332599 learned_addr learned my addr 172.21.15.129:0/3897332599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:38.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.498+0000 7ff43a5ab640 1 -- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff43415d760 msgr2=0x7ff4340cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.498+0000 7ff43a5ab640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff43415d760 0x7ff4340cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.498+0000 7ff43a5ab640 1 -- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff43406c0d0 msgr2=0x7ff4341f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.498+0000 7ff43a5ab640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff43406c0d0 0x7ff4341f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.498+0000 7ff43a5ab640 1 -- 172.21.15.129:0/3897332599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff424009400 con 0x7ff43415db40 2024-03-23T16:01:38.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.500+0000 7ff43a5ab640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff4341e55f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00caa0 tx=0x7ff42c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.504+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00aae0 con 0x7ff43415db40 2024-03-23T16:01:38.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.504+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42c00ac40 con 0x7ff43415db40 2024-03-23T16:01:38.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.504+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c012400 con 0x7ff43415db40 2024-03-23T16:01:38.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.504+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4340c71f0 con 0x7ff43415db40 2024-03-23T16:01:38.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.504+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4340c77b0 con 0x7ff43415db40 2024-03-23T16:01:38.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.505+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4340cdee0 con 0x7ff43415db40 2024-03-23T16:01:38.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.525+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff42c007070 con 0x7ff43415db40 2024-03-23T16:01:38.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.526+0000 7ff42a7fc640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff41806b810 0x7ff41806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.526+0000 7ff439daa640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff41806b810 0x7ff41806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.527+0000 7ff439daa640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff41806b810 0x7ff41806dcd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff430009990 tx=0x7ff430009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.529+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff42c058e10 con 0x7ff43415db40 2024-03-23T16:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.529+0000 7ff42a7fc640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7ff418073b10 0x7ff418075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.529+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 --> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff418076600 con 0x7ff418073b10 2024-03-23T16:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.529+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff42c0019f0 con 0x7ff43415db40 2024-03-23T16:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.530+0000 7ff4395a9640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7ff418073b10 0x7ff418075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.530+0000 7ff4395a9640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7ff418073b10 0x7ff418075f30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.531+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== osd.7 v2:172.21.15.157:6818/2526630177 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff418076600 con 0x7ff418073b10 2024-03-23T16:01:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.544+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 --> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff4340cdc00 con 0x7ff418073b10 2024-03-23T16:01:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff42a7fc640 1 -- 172.21.15.129:0/3897332599 <== osd.7 v2:172.21.15.157:6818/2526630177 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff424031b80 con 0x7ff418073b10 2024-03-23T16:01:38.547 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-23T16:01:38.547 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T16:01:38.547 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-23T16:01:38.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7ff418073b10 msgr2=0x7ff418075f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6818/2526630177,v1:172.21.15.157:6819/2526630177] conn(0x7ff418073b10 0x7ff418075f30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff41806b810 msgr2=0x7ff41806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff41806b810 0x7ff41806dcd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff430009990 tx=0x7ff430009110 comp rx=0 tx=0).stop 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 msgr2=0x7ff4341e55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 --2- 172.21.15.129:0/3897332599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff43415db40 0x7ff4341e55f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00caa0 tx=0x7ff42c00cf60 comp rx=0 tx=0).stop 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43a5ab640 1 -- 172.21.15.129:0/3897332599 reap_dead start 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 shutdown_connections 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 >> 172.21.15.129:0/3897332599 conn(0x7ff4340b1630 msgr2=0x7ff4340b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 shutdown_connections 2024-03-23T16:01:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.545+0000 7ff43c00e640 1 -- 172.21.15.129:0/3897332599 wait complete. 2024-03-23T16:01:38.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-23T16:01:38.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.660+0000 7f172fcf4640 1 Processor -- start 2024-03-23T16:01:38.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.660+0000 7f172fcf4640 1 -- start start 2024-03-23T16:01:38.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.661+0000 7f172fcf4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f172815aea0 0x7f17280ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.661+0000 7f172fcf4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17280ce990 0x7f17280cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.661+0000 7f172fcf4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ccf40 0x7f172806d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.661+0000 7f172fcf4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f1728154bc0 con 0x7f172815aea0 2024-03-23T16:01:38.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.661+0000 7f172fcf4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f17280cfa30 con 0x7f17280ce990 2024-03-23T16:01:38.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.661+0000 7f172fcf4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17280cf0e0 con 0x7f17280ccf40 2024-03-23T16:01:38.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.662+0000 7f172da90640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f172815aea0 0x7f17280ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.662+0000 7f172d28f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17280ce990 0x7f17280cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.662+0000 7f172e291640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ccf40 0x7f172806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.662+0000 7f172da90640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f172815aea0 0x7f17280ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33810/0 (socket says 172.21.15.129:33810) 2024-03-23T16:01:38.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.662+0000 7f172da90640 1 -- 172.21.15.129:0/1100387449 learned_addr learned my addr 172.21.15.129:0/1100387449 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:38.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.664+0000 7f172da90640 1 -- 172.21.15.129:0/1100387449 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ccf40 msgr2=0x7f172806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:38.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.664+0000 7f172da90640 1 --2- 172.21.15.129:0/1100387449 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ccf40 0x7f172806d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.664+0000 7f172da90640 1 -- 172.21.15.129:0/1100387449 >> v2:172.21.15.157:3300/0 conn(0x7f17280ce990 msgr2=0x7f17280cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:38.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.664+0000 7f172da90640 1 --2- 172.21.15.129:0/1100387449 >> v2:172.21.15.157:3300/0 conn(0x7f17280ce990 0x7f17280cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.664+0000 7f172da90640 1 -- 172.21.15.129:0/1100387449 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17280cdb50 con 0x7f172815aea0 2024-03-23T16:01:38.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.667+0000 7f172da90640 1 --2- 172.21.15.129:0/1100387449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172815aea0 0x7f17280ce390 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f172806cd30 tx=0x7f172402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3d15c56bcec765 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172ca8e640 1 -- 172.21.15.129:0/1100387449 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17240317e0 con 0x7f172815aea0 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172ca8e640 1 -- 172.21.15.129:0/1100387449 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1724031940 con 0x7f172815aea0 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172ca8e640 1 -- 172.21.15.129:0/1100387449 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724031cd0 con 0x7f172815aea0 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 -- 172.21.15.129:0/1100387449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172815aea0 msgr2=0x7f17280ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 --2- 172.21.15.129:0/1100387449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172815aea0 0x7f17280ce390 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f172806cd30 tx=0x7f172402b880 comp rx=0 tx=0).stop 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 -- 172.21.15.129:0/1100387449 shutdown_connections 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 --2- 172.21.15.129:0/1100387449 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ccf40 0x7f172806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 --2- 172.21.15.129:0/1100387449 >> v2:172.21.15.157:3300/0 conn(0x7f17280ce990 0x7f17280cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 --2- 172.21.15.129:0/1100387449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f172815aea0 0x7f17280ce390 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 -- 172.21.15.129:0/1100387449 >> 172.21.15.129:0/1100387449 conn(0x7f17280cfc20 msgr2=0x7f17280d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 -- 172.21.15.129:0/1100387449 shutdown_connections 2024-03-23T16:01:38.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.671+0000 7f172fcf4640 1 -- 172.21.15.129:0/1100387449 wait complete. 2024-03-23T16:01:38.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 Processor -- start 2024-03-23T16:01:38.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 -- start start 2024-03-23T16:01:38.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17280ccf40 0x7f1728157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ce990 0x7f1728157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172815aea0 0x7f1728152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17281f3da0 con 0x7f17280ccf40 2024-03-23T16:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17281f3820 con 0x7f172815aea0 2024-03-23T16:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.673+0000 7f172fcf4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17281f3ac0 con 0x7f17280ce990 2024-03-23T16:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.674+0000 7f172e291640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172815aea0 0x7f1728152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.674+0000 7f172da90640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17280ccf40 0x7f1728157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.674+0000 7f172d28f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ce990 0x7f1728157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.674+0000 7f172e291640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172815aea0 0x7f1728152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:45102/0 (socket says 172.21.15.129:45102) 2024-03-23T16:01:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.674+0000 7f172e291640 1 -- 172.21.15.129:0/3022333613 learned_addr learned my addr 172.21.15.129:0/3022333613 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.678+0000 7f172da90640 1 -- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ce990 msgr2=0x7f1728157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.678+0000 7f172da90640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ce990 0x7f1728157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.678+0000 7f172da90640 1 -- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172815aea0 msgr2=0x7f1728152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.678+0000 7f172da90640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172815aea0 0x7f1728152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.678+0000 7f172da90640 1 -- 172.21.15.129:0/3022333613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17240094d0 con 0x7f17280ccf40 2024-03-23T16:01:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.680+0000 7f172da90640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17280ccf40 0x7f1728157110 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1724033000 tx=0x7f17240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.684+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724032670 con 0x7f17280ccf40 2024-03-23T16:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.684+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1724032810 con 0x7f17280ccf40 2024-03-23T16:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.684+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17280c38f0 con 0x7f17280ccf40 2024-03-23T16:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.684+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724032bc0 con 0x7f17280ccf40 2024-03-23T16:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.684+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17280c3eb0 con 0x7f17280ccf40 2024-03-23T16:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.685+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17280c3750 con 0x7f17280ccf40 2024-03-23T16:01:38.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.950+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1724032d20 con 0x7f17280ccf40 2024-03-23T16:01:38.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.950+0000 7f171e7fc640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f170c06b840 0x7f170c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:38.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.950+0000 7f172d28f640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f170c06b840 0x7f170c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:38.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.951+0000 7f172d28f640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f170c06b840 0x7f170c06dd00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f17280c6c50 tx=0x7f1718009110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:38.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.968+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f172407a900 con 0x7f17280ccf40 2024-03-23T16:01:38.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:38.968+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1724076ba0 con 0x7f17280ccf40 2024-03-23T16:01:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.153+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f17280cddf0 con 0x7f170c06b840 2024-03-23T16:01:39.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.153+0000 7f171e7fc640 1 -- 172.21.15.129:0/3022333613 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1718004020 con 0x7f170c06b840 2024-03-23T16:01:39.155 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 7 to deep-scrub 2024-03-23T16:01:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f170c06b840 msgr2=0x7f170c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f170c06b840 0x7f170c06dd00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f17280c6c50 tx=0x7f1718009110 comp rx=0 tx=0).stop 2024-03-23T16:01:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17280ccf40 msgr2=0x7f1728157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17280ccf40 0x7f1728157110 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1724033000 tx=0x7f17240046c0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 shutdown_connections 2024-03-23T16:01:39.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f170c06b840 0x7f170c06dd00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f172815aea0 0x7f1728152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17280ce990 0x7f1728157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 --2- 172.21.15.129:0/3022333613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17280ccf40 0x7f1728157110 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 >> 172.21.15.129:0/3022333613 conn(0x7f17280cfc20 msgr2=0x7f1728074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 shutdown_connections 2024-03-23T16:01:39.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.155+0000 7f172fcf4640 1 -- 172.21.15.129:0/3022333613 wait complete. 2024-03-23T16:01:39.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T16:01:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.271+0000 7fb135719640 1 Processor -- start 2024-03-23T16:01:39.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.271+0000 7fb135719640 1 -- start start 2024-03-23T16:01:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.272+0000 7fb135719640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb13015a2e0 0x7fb13006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.272+0000 7fb135719640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb13006c0b0 0x7fb1300c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.272+0000 7fb135719640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1300cbcb0 con 0x7fb1301544b0 2024-03-23T16:01:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.272+0000 7fb135719640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7fb1300c5be0 con 0x7fb13006c0b0 2024-03-23T16:01:39.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.272+0000 7fb135719640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb13015b280 con 0x7fb13015a2e0 2024-03-23T16:01:39.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.278+0000 7fb12f7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb13006c0b0 0x7fb1300c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.279+0000 7fb12f7fe640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7fb13006c0b0 0x7fb1300c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33826/0 (socket says 172.21.15.129:33826) 2024-03-23T16:01:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.279+0000 7fb12f7fe640 1 -- 172.21.15.129:0/456166172 learned_addr learned my addr 172.21.15.129:0/456166172 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.283+0000 7fb12f7fe640 1 -- 172.21.15.129:0/456166172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb13015a2e0 msgr2=0x7fb13006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.283+0000 7fb12f7fe640 1 --2- 172.21.15.129:0/456166172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb13015a2e0 0x7fb13006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.283+0000 7fb12f7fe640 1 -- 172.21.15.129:0/456166172 >> v1:172.21.15.157:6789/0 conn(0x7fb1301544b0 legacy=0x7fb1300caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.283+0000 7fb12f7fe640 1 -- 172.21.15.129:0/456166172 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13015a9e0 con 0x7fb13006c0b0 2024-03-23T16:01:39.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.285+0000 7fb12f7fe640 1 --2- 172.21.15.129:0/456166172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb1300c6000 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb128009e50 tx=0x7fb12802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbf9ae8dbabf8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:39.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb12dffb640 1 -- 172.21.15.129:0/456166172 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12802cb00 con 0x7fb13006c0b0 2024-03-23T16:01:39.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb12dffb640 1 -- 172.21.15.129:0/456166172 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12802cc60 con 0x7fb13006c0b0 2024-03-23T16:01:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb12dffb640 1 -- 172.21.15.129:0/456166172 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1280343f0 con 0x7fb13006c0b0 2024-03-23T16:01:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 -- 172.21.15.129:0/456166172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 msgr2=0x7fb1300c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 --2- 172.21.15.129:0/456166172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb1300c6000 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb128009e50 tx=0x7fb12802bf60 comp rx=0 tx=0).stop 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 -- 172.21.15.129:0/456166172 shutdown_connections 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 --2- 172.21.15.129:0/456166172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb1300c6000 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 --2- 172.21.15.129:0/456166172 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb13015a2e0 0x7fb13006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 -- 172.21.15.129:0/456166172 >> 172.21.15.129:0/456166172 conn(0x7fb130061b50 msgr2=0x7fb130061f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 -- 172.21.15.129:0/456166172 shutdown_connections 2024-03-23T16:01:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.291+0000 7fb135719640 1 -- 172.21.15.129:0/456166172 wait complete. 2024-03-23T16:01:39.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.293+0000 7fb135719640 1 Processor -- start 2024-03-23T16:01:39.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.293+0000 7fb135719640 1 -- start start 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.293+0000 7fb135719640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb13014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb135719640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1301544b0 0x7fb130141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb135719640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb13015a2e0 0x7fb13014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb135719640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300cf5b0 con 0x7fb13006c0b0 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb135719640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300d0360 con 0x7fb13015a2e0 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb135719640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb13014c920 con 0x7fb1301544b0 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb12effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb13014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb12effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb13014c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33830/0 (socket says 172.21.15.129:33830) 2024-03-23T16:01:39.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.294+0000 7fb12effd640 1 -- 172.21.15.129:0/411865783 learned_addr learned my addr 172.21.15.129:0/411865783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.298+0000 7fb12effd640 1 -- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1301544b0 msgr2=0x7fb130141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.298+0000 7fb12effd640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1301544b0 0x7fb130141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.298+0000 7fb12effd640 1 -- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb13015a2e0 msgr2=0x7fb13014d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.298+0000 7fb12effd640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb13015a2e0 0x7fb13014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.298+0000 7fb12effd640 1 -- 172.21.15.129:0/411865783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb128009b30 con 0x7fb13006c0b0 2024-03-23T16:01:39.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.300+0000 7fb12effd640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb13014c3e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb12000c410 tx=0x7fb12000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:39.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.304+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1200035f0 con 0x7fb13006c0b0 2024-03-23T16:01:39.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.304+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb120003770 con 0x7fb13006c0b0 2024-03-23T16:01:39.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.304+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120003b20 con 0x7fb13006c0b0 2024-03-23T16:01:39.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.304+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13014dfa0 con 0x7fb13006c0b0 2024-03-23T16:01:39.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.304+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1300cf800 con 0x7fb13006c0b0 2024-03-23T16:01:39.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.306+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb130075740 con 0x7fb13006c0b0 2024-03-23T16:01:39.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.325+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb120003c80 con 0x7fb13006c0b0 2024-03-23T16:01:39.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.325+0000 7fb1277fe640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb11006b810 0x7fb11006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:39.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.326+0000 7fb12e7fc640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb11006b810 0x7fb11006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.326+0000 7fb12e7fc640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb11006b810 0x7fb11006dcd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb130145450 tx=0x7fb118006bf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.328+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7fb120058b20 con 0x7fb13006c0b0 2024-03-23T16:01:39.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:01:39.336 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:01:39.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.571+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1200546c0 con 0x7fb13006c0b0 2024-03-23T16:01:39.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.783+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb1301531e0 con 0x7fb11006b810 2024-03-23T16:01:39.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.795+0000 7fb1277fe640 1 -- 172.21.15.129:0/411865783 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250415 (secure 0 0 0) 0x7fb1301531e0 con 0x7fb11006b810 2024-03-23T16:01:39.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:01:39.801 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb11006b810 msgr2=0x7fb11006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb11006b810 0x7fb11006dcd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb130145450 tx=0x7fb118006bf0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 msgr2=0x7fb13014c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb13014c3e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb12000c410 tx=0x7fb12000c8d0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 shutdown_connections 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7fb11006b810 0x7fb11006dcd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb13015a2e0 0x7fb13014d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1301544b0 0x7fb130141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 --2- 172.21.15.129:0/411865783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13006c0b0 0x7fb13014c3e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 >> 172.21.15.129:0/411865783 conn(0x7fb130061b50 msgr2=0x7fb130151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 shutdown_connections 2024-03-23T16:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:39.799+0000 7fb135719640 1 -- 172.21.15.129:0/411865783 wait complete. 2024-03-23T16:01:39.812 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-03-23T16:01:38.585438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":324,"ondisk_log_size":324,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235156,"kb_used_data":2368,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739564,"statfs":{"total":773094113280,"available":772853313536,"internally_reserved":0,"allocated":2424832,"data_stored":1948916,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.377823+0000","last_change":"2024-03-23T15:47:27.453476+0000","last_active":"2024-03-23T15:59:56.377823+0000","last_peered":"2024-03-23T15:59:56.377823+0000","last_clean":"2024-03-23T15:59:56.377823+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T15:59:56.377823+0000","last_undegraded":"2024-03-23T15:59:56.377823+0000","last_fullsized":"2024-03-23T15:59:56.377823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298822+0000","last_change":"2024-03-23T15:47:27.382930+0000","last_active":"2024-03-23T15:57:51.298822+0000","last_peered":"2024-03-23T15:57:51.298822+0000","last_clean":"2024-03-23T15:57:51.298822+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T15:57:51.298822+0000","last_undegraded":"2024-03-23T15:57:51.298822+0000","last_fullsized":"2024-03-23T15:57:51.298822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:46.005464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298806+0000","last_change":"2024-03-23T15:47:27.439007+0000","last_active":"2024-03-23T15:57:51.298806+0000","last_peered":"2024-03-23T15:57:51.298806+0000","last_clean":"2024-03-23T15:57:51.298806+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T15:57:51.298806+0000","last_undegraded":"2024-03-23T15:57:51.298806+0000","last_fullsized":"2024-03-23T15:57:51.298806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:46.371731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.438780+0000","last_change":"2024-03-23T15:47:27.449748+0000","last_active":"2024-03-23T16:01:26.438780+0000","last_peered":"2024-03-23T16:01:26.438780+0000","last_clean":"2024-03-23T16:01:26.438780+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T16:01:26.438780+0000","last_undegraded":"2024-03-23T16:01:26.438780+0000","last_fullsized":"2024-03-23T16:01:26.438780+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.377765+0000","last_change":"2024-03-23T15:47:27.382879+0000","last_active":"2024-03-23T15:59:56.377765+0000","last_peered":"2024-03-23T15:59:56.377765+0000","last_clean":"2024-03-23T15:59:56.377765+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T15:59:56.377765+0000","last_undegraded":"2024-03-23T15:59:56.377765+0000","last_fullsized":"2024-03-23T15:59:56.377765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.432118+0000","last_change":"2024-03-23T15:47:27.443245+0000","last_active":"2024-03-23T15:59:56.432118+0000","last_peered":"2024-03-23T15:59:56.432118+0000","last_clean":"2024-03-23T15:59:56.432118+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T15:59:56.432118+0000","last_undegraded":"2024-03-23T15:59:56.432118+0000","last_fullsized":"2024-03-23T15:59:56.432118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.377715+0000","last_change":"2024-03-23T15:47:27.438718+0000","last_active":"2024-03-23T15:59:56.377715+0000","last_peered":"2024-03-23T15:59:56.377715+0000","last_clean":"2024-03-23T15:59:56.377715+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T15:59:56.377715+0000","last_undegraded":"2024-03-23T15:59:56.377715+0000","last_fullsized":"2024-03-23T15:59:56.377715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296437+0000","last_change":"2024-03-23T15:47:27.384014+0000","last_active":"2024-03-23T15:57:26.296437+0000","last_peered":"2024-03-23T15:57:26.296437+0000","last_clean":"2024-03-23T15:57:26.296437+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T15:57:26.296437+0000","last_undegraded":"2024-03-23T15:57:26.296437+0000","last_fullsized":"2024-03-23T15:57:26.296437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:14:46.187983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:31.450477+0000","last_change":"2024-03-23T15:47:27.456425+0000","last_active":"2024-03-23T15:59:31.450477+0000","last_peered":"2024-03-23T15:59:31.450477+0000","last_clean":"2024-03-23T15:59:31.450477+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T15:59:31.450477+0000","last_undegraded":"2024-03-23T15:59:31.450477+0000","last_fullsized":"2024-03-23T15:59:31.450477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:31.375388+0000","last_change":"2024-03-23T15:47:27.452163+0000","last_active":"2024-03-23T15:59:31.375388+0000","last_peered":"2024-03-23T15:59:31.375388+0000","last_clean":"2024-03-23T15:59:31.375388+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T15:59:31.375388+0000","last_undegraded":"2024-03-23T15:59:31.375388+0000","last_fullsized":"2024-03-23T15:59:31.375388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:46.377367+0000","last_change":"2024-03-23T15:48:04.931538+0000","last_active":"2024-03-23T15:57:46.377367+0000","last_peered":"2024-03-23T15:57:46.377367+0000","last_clean":"2024-03-23T15:57:46.377367+0000","last_became_active":"2024-03-23T15:48:04.931241+0000","last_became_peered":"2024-03-23T15:48:04.931241+0000","last_unstale":"2024-03-23T15:57:46.377367+0000","last_undegraded":"2024-03-23T15:57:46.377367+0000","last_fullsized":"2024-03-23T15:57:46.377367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:31.375354+0000","last_change":"2024-03-23T15:47:27.452711+0000","last_active":"2024-03-23T15:59:31.375354+0000","last_peered":"2024-03-23T15:59:31.375354+0000","last_clean":"2024-03-23T15:59:31.375354+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T15:59:31.375354+0000","last_undegraded":"2024-03-23T15:59:31.375354+0000","last_fullsized":"2024-03-23T15:59:31.375354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356105+0000","last_change":"2024-03-23T15:47:27.355873+0000","last_active":"2024-03-23T15:57:06.356105+0000","last_peered":"2024-03-23T15:57:06.356105+0000","last_clean":"2024-03-23T15:57:06.356105+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T15:57:06.356105+0000","last_undegraded":"2024-03-23T15:57:06.356105+0000","last_fullsized":"2024-03-23T15:57:06.356105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:20:39.126348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366410+0000","last_change":"2024-03-23T15:47:27.449076+0000","last_active":"2024-03-23T15:59:06.366410+0000","last_peered":"2024-03-23T15:59:06.366410+0000","last_clean":"2024-03-23T15:59:06.366410+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T15:59:06.366410+0000","last_undegraded":"2024-03-23T15:59:06.366410+0000","last_fullsized":"2024-03-23T15:59:06.366410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:16:37.878605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.305751+0000","last_change":"2024-03-23T15:47:27.383728+0000","last_active":"2024-03-23T15:59:06.305751+0000","last_peered":"2024-03-23T15:59:06.305751+0000","last_clean":"2024-03-23T15:59:06.305751+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T15:59:06.305751+0000","last_undegraded":"2024-03-23T15:59:06.305751+0000","last_fullsized":"2024-03-23T15:59:06.305751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.431850+0000","last_change":"2024-03-23T15:47:27.456573+0000","last_active":"2024-03-23T16:01:26.431850+0000","last_peered":"2024-03-23T16:01:26.431850+0000","last_clean":"2024-03-23T16:01:26.431850+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T16:01:26.431850+0000","last_undegraded":"2024-03-23T16:01:26.431850+0000","last_fullsized":"2024-03-23T16:01:26.431850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.170661+0000","last_change":"2024-03-23T16:01:36.170703+0000","last_active":"2024-03-23T16:01:36.170661+0000","last_peered":"2024-03-23T16:01:36.170661+0000","last_clean":"2024-03-23T16:01:36.170661+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T16:01:36.170661+0000","last_undegraded":"2024-03-23T16:01:36.170661+0000","last_fullsized":"2024-03-23T16:01:36.170661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:36.170584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:36.170584+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.170584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:20:46.816752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383777+0000","last_change":"2024-03-23T15:47:27.452881+0000","last_active":"2024-03-23T15:59:06.383777+0000","last_peered":"2024-03-23T15:59:06.383777+0000","last_clean":"2024-03-23T15:59:06.383777+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T15:59:06.383777+0000","last_undegraded":"2024-03-23T15:59:06.383777+0000","last_fullsized":"2024-03-23T15:59:06.383777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:04.963424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.429362+0000","last_change":"2024-03-23T15:47:27.454576+0000","last_active":"2024-03-23T16:00:56.429362+0000","last_peered":"2024-03-23T16:00:56.429362+0000","last_clean":"2024-03-23T16:00:56.429362+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T16:00:56.429362+0000","last_undegraded":"2024-03-23T16:00:56.429362+0000","last_fullsized":"2024-03-23T16:00:56.429362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.382962+0000","last_change":"2024-03-23T15:47:27.452026+0000","last_active":"2024-03-23T16:00:56.382962+0000","last_peered":"2024-03-23T16:00:56.382962+0000","last_clean":"2024-03-23T16:00:56.382962+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T16:00:56.382962+0000","last_undegraded":"2024-03-23T16:00:56.382962+0000","last_fullsized":"2024-03-23T16:00:56.382962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446916+0000","last_change":"2024-03-23T15:47:27.449279+0000","last_active":"2024-03-23T15:58:41.446916+0000","last_peered":"2024-03-23T15:58:41.446916+0000","last_clean":"2024-03-23T15:58:41.446916+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T15:58:41.446916+0000","last_undegraded":"2024-03-23T15:58:41.446916+0000","last_fullsized":"2024-03-23T15:58:41.446916+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:20.051008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.457273+0000","last_change":"2024-03-23T15:47:27.454978+0000","last_active":"2024-03-23T16:00:56.457273+0000","last_peered":"2024-03-23T16:00:56.457273+0000","last_clean":"2024-03-23T16:00:56.457273+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T16:00:56.457273+0000","last_undegraded":"2024-03-23T16:00:56.457273+0000","last_fullsized":"2024-03-23T16:00:56.457273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303312+0000","last_change":"2024-03-23T15:47:27.450996+0000","last_active":"2024-03-23T15:58:41.303312+0000","last_peered":"2024-03-23T15:58:41.303312+0000","last_clean":"2024-03-23T15:58:41.303312+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T15:58:41.303312+0000","last_undegraded":"2024-03-23T15:58:41.303312+0000","last_fullsized":"2024-03-23T15:58:41.303312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:26.426549+0000","last_change":"2024-03-23T15:47:27.451947+0000","last_active":"2024-03-23T16:00:26.426549+0000","last_peered":"2024-03-23T16:00:26.426549+0000","last_clean":"2024-03-23T16:00:26.426549+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T16:00:26.426549+0000","last_undegraded":"2024-03-23T16:00:26.426549+0000","last_fullsized":"2024-03-23T16:00:26.426549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.301061+0000","last_change":"2024-03-23T15:47:27.450281+0000","last_active":"2024-03-23T15:58:16.301061+0000","last_peered":"2024-03-23T15:58:16.301061+0000","last_clean":"2024-03-23T15:58:16.301061+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T15:58:16.301061+0000","last_undegraded":"2024-03-23T15:58:16.301061+0000","last_fullsized":"2024-03-23T15:58:16.301061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:47.440708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:26.434458+0000","last_change":"2024-03-23T15:47:27.456817+0000","last_active":"2024-03-23T16:00:26.434458+0000","last_peered":"2024-03-23T16:00:26.434458+0000","last_clean":"2024-03-23T16:00:26.434458+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T16:00:26.434458+0000","last_undegraded":"2024-03-23T16:00:26.434458+0000","last_fullsized":"2024-03-23T16:00:26.434458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362212+0000","last_change":"2024-03-23T15:47:27.448426+0000","last_active":"2024-03-23T15:58:16.362212+0000","last_peered":"2024-03-23T15:58:16.362212+0000","last_clean":"2024-03-23T15:58:16.362212+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T15:58:16.362212+0000","last_undegraded":"2024-03-23T15:58:16.362212+0000","last_fullsized":"2024-03-23T15:58:16.362212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362193+0000","last_change":"2024-03-23T15:47:27.355811+0000","last_active":"2024-03-23T15:58:16.362193+0000","last_peered":"2024-03-23T15:58:16.362193+0000","last_clean":"2024-03-23T15:58:16.362193+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T15:58:16.362193+0000","last_undegraded":"2024-03-23T15:58:16.362193+0000","last_fullsized":"2024-03-23T15:58:16.362193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:21.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298768+0000","last_change":"2024-03-23T15:47:27.450677+0000","last_active":"2024-03-23T15:57:51.298768+0000","last_peered":"2024-03-23T15:57:51.298768+0000","last_clean":"2024-03-23T15:57:51.298768+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T15:57:51.298768+0000","last_undegraded":"2024-03-23T15:57:51.298768+0000","last_fullsized":"2024-03-23T15:57:51.298768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:01.229752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.424007+0000","last_change":"2024-03-23T15:47:27.452391+0000","last_active":"2024-03-23T15:59:56.424007+0000","last_peered":"2024-03-23T15:59:56.424007+0000","last_clean":"2024-03-23T15:59:56.424007+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T15:59:56.424007+0000","last_undegraded":"2024-03-23T15:59:56.424007+0000","last_fullsized":"2024-03-23T15:59:56.424007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377740+0000","last_change":"2024-03-23T15:47:27.455718+0000","last_active":"2024-03-23T15:57:51.377740+0000","last_peered":"2024-03-23T15:57:51.377740+0000","last_clean":"2024-03-23T15:57:51.377740+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T15:57:51.377740+0000","last_undegraded":"2024-03-23T15:57:51.377740+0000","last_fullsized":"2024-03-23T15:57:51.377740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:06.863197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.432092+0000","last_change":"2024-03-23T15:47:27.443127+0000","last_active":"2024-03-23T15:59:56.432092+0000","last_peered":"2024-03-23T15:59:56.432092+0000","last_clean":"2024-03-23T15:59:56.432092+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T15:59:56.432092+0000","last_undegraded":"2024-03-23T15:59:56.432092+0000","last_fullsized":"2024-03-23T15:59:56.432092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.452324+0000","last_change":"2024-03-23T15:47:27.448869+0000","last_active":"2024-03-23T15:59:56.452324+0000","last_peered":"2024-03-23T15:59:56.452324+0000","last_clean":"2024-03-23T15:59:56.452324+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T15:59:56.452324+0000","last_undegraded":"2024-03-23T15:59:56.452324+0000","last_fullsized":"2024-03-23T15:59:56.452324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.296495+0000","last_change":"2024-03-23T15:47:26.151546+0000","last_active":"2024-03-23T15:57:26.296495+0000","last_peered":"2024-03-23T15:57:26.296495+0000","last_clean":"2024-03-23T15:57:26.296495+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T15:57:26.296495+0000","last_undegraded":"2024-03-23T15:57:26.296495+0000","last_fullsized":"2024-03-23T15:57:26.296495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:26.440553+0000","last_change":"2024-03-23T15:47:27.438806+0000","last_active":"2024-03-23T15:57:26.440553+0000","last_peered":"2024-03-23T15:57:26.440553+0000","last_clean":"2024-03-23T15:57:26.440553+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T15:57:26.440553+0000","last_undegraded":"2024-03-23T15:57:26.440553+0000","last_fullsized":"2024-03-23T15:57:26.440553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:00:55.988206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.424045+0000","last_change":"2024-03-23T15:47:26.151251+0000","last_active":"2024-03-23T15:59:56.424045+0000","last_peered":"2024-03-23T15:59:56.424045+0000","last_clean":"2024-03-23T15:59:56.424045+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T15:59:56.424045+0000","last_undegraded":"2024-03-23T15:59:56.424045+0000","last_fullsized":"2024-03-23T15:59:56.424045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.356066+0000","last_change":"2024-03-23T15:47:26.151505+0000","last_active":"2024-03-23T15:57:06.356066+0000","last_peered":"2024-03-23T15:57:06.356066+0000","last_clean":"2024-03-23T15:57:06.356066+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T15:57:06.356066+0000","last_undegraded":"2024-03-23T15:57:06.356066+0000","last_fullsized":"2024-03-23T15:57:06.356066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294507+0000","last_change":"2024-03-23T15:47:27.384435+0000","last_active":"2024-03-23T15:57:06.294507+0000","last_peered":"2024-03-23T15:57:06.294507+0000","last_clean":"2024-03-23T15:57:06.294507+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T15:57:06.294507+0000","last_undegraded":"2024-03-23T15:57:06.294507+0000","last_fullsized":"2024-03-23T15:57:06.294507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:38:14.371576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:31.421630+0000","last_change":"2024-03-23T15:47:26.147740+0000","last_active":"2024-03-23T15:59:31.421630+0000","last_peered":"2024-03-23T15:59:31.421630+0000","last_clean":"2024-03-23T15:59:31.421630+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T15:59:31.421630+0000","last_undegraded":"2024-03-23T15:59:31.421630+0000","last_fullsized":"2024-03-23T15:59:31.421630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.439044+0000","last_change":"2024-03-23T15:47:27.450935+0000","last_active":"2024-03-23T15:57:06.439044+0000","last_peered":"2024-03-23T15:57:06.439044+0000","last_clean":"2024-03-23T15:57:06.439044+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T15:57:06.439044+0000","last_undegraded":"2024-03-23T15:57:06.439044+0000","last_fullsized":"2024-03-23T15:57:06.439044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"24'7","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:35.753403+0000","last_change":"2024-03-23T16:01:35.753452+0000","last_active":"2024-03-23T16:01:35.753403+0000","last_peered":"2024-03-23T16:01:35.753403+0000","last_clean":"2024-03-23T16:01:35.753403+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T16:01:35.753403+0000","last_undegraded":"2024-03-23T16:01:35.753403+0000","last_fullsized":"2024-03-23T16:01:35.753403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-23T16:01:35.753304+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-23T16:01:35.753304+0000","last_clean_scrub_stamp":"2024-03-23T16:01:35.753304+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:06:57.213380+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:31.429964+0000","last_change":"2024-03-23T15:47:27.456157+0000","last_active":"2024-03-23T15:59:31.429964+0000","last_peered":"2024-03-23T15:59:31.429964+0000","last_clean":"2024-03-23T15:59:31.429964+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T15:59:31.429964+0000","last_undegraded":"2024-03-23T15:59:31.429964+0000","last_fullsized":"2024-03-23T15:59:31.429964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"24'6","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.336134+0000","last_change":"2024-03-23T16:01:36.336158+0000","last_active":"2024-03-23T16:01:36.336134+0000","last_peered":"2024-03-23T16:01:36.336134+0000","last_clean":"2024-03-23T16:01:36.336134+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T16:01:36.336134+0000","last_undegraded":"2024-03-23T16:01:36.336134+0000","last_fullsized":"2024-03-23T16:01:36.336134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-23T16:01:36.336071+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-23T16:01:36.336071+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.336071+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:59.170006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.294544+0000","last_change":"2024-03-23T15:47:27.454697+0000","last_active":"2024-03-23T15:57:06.294544+0000","last_peered":"2024-03-23T15:57:06.294544+0000","last_clean":"2024-03-23T15:57:06.294544+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T15:57:06.294544+0000","last_undegraded":"2024-03-23T15:57:06.294544+0000","last_fullsized":"2024-03-23T15:57:06.294544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:28.733809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:06.374127+0000","last_change":"2024-03-23T15:47:26.147175+0000","last_active":"2024-03-23T15:57:06.374127+0000","last_peered":"2024-03-23T15:57:06.374127+0000","last_clean":"2024-03-23T15:57:06.374127+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T15:57:06.374127+0000","last_undegraded":"2024-03-23T15:57:06.374127+0000","last_fullsized":"2024-03-23T15:57:06.374127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:22:06.166339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.431964+0000","last_change":"2024-03-23T15:47:27.455220+0000","last_active":"2024-03-23T16:01:26.431964+0000","last_peered":"2024-03-23T16:01:26.431964+0000","last_clean":"2024-03-23T16:01:26.431964+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T16:01:26.431964+0000","last_undegraded":"2024-03-23T16:01:26.431964+0000","last_fullsized":"2024-03-23T16:01:26.431964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"24'16","reported_seq":51,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.065500+0000","last_change":"2024-03-23T16:01:36.065535+0000","last_active":"2024-03-23T16:01:36.065500+0000","last_peered":"2024-03-23T16:01:36.065500+0000","last_clean":"2024-03-23T16:01:36.065500+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T16:01:36.065500+0000","last_undegraded":"2024-03-23T16:01:36.065500+0000","last_fullsized":"2024-03-23T16:01:36.065500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'16","last_scrub_stamp":"2024-03-23T16:01:36.065425+0000","last_deep_scrub":"24'16","last_deep_scrub_stamp":"2024-03-23T16:01:36.065425+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.065425+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:06:57.525502+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:31.375378+0000","last_change":"2024-03-23T15:47:27.356681+0000","last_active":"2024-03-23T15:59:31.375378+0000","last_peered":"2024-03-23T15:59:31.375378+0000","last_clean":"2024-03-23T15:59:31.375378+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T15:59:31.375378+0000","last_undegraded":"2024-03-23T15:59:31.375378+0000","last_fullsized":"2024-03-23T15:59:31.375378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.383756+0000","last_change":"2024-03-23T15:47:26.152254+0000","last_active":"2024-03-23T15:59:06.383756+0000","last_peered":"2024-03-23T15:59:06.383756+0000","last_clean":"2024-03-23T15:59:06.383756+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T15:59:06.383756+0000","last_undegraded":"2024-03-23T15:59:06.383756+0000","last_fullsized":"2024-03-23T15:59:06.383756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.366429+0000","last_change":"2024-03-23T15:47:27.456350+0000","last_active":"2024-03-23T15:59:06.366429+0000","last_peered":"2024-03-23T15:59:06.366429+0000","last_clean":"2024-03-23T15:59:06.366429+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T15:59:06.366429+0000","last_undegraded":"2024-03-23T15:59:06.366429+0000","last_fullsized":"2024-03-23T15:59:06.366429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:49:56.951166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.431928+0000","last_change":"2024-03-23T15:47:26.146616+0000","last_active":"2024-03-23T16:01:26.431928+0000","last_peered":"2024-03-23T16:01:26.431928+0000","last_clean":"2024-03-23T16:01:26.431928+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T16:01:26.431928+0000","last_undegraded":"2024-03-23T16:01:26.431928+0000","last_fullsized":"2024-03-23T16:01:26.431928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.431947+0000","last_change":"2024-03-23T15:47:27.451730+0000","last_active":"2024-03-23T16:01:26.431947+0000","last_peered":"2024-03-23T16:01:26.431947+0000","last_clean":"2024-03-23T16:01:26.431947+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T16:01:26.431947+0000","last_undegraded":"2024-03-23T16:01:26.431947+0000","last_fullsized":"2024-03-23T16:01:26.431947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.385328+0000","last_change":"2024-03-23T15:47:26.153645+0000","last_active":"2024-03-23T16:01:26.385328+0000","last_peered":"2024-03-23T16:01:26.385328+0000","last_clean":"2024-03-23T16:01:26.385328+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T16:01:26.385328+0000","last_undegraded":"2024-03-23T16:01:26.385328+0000","last_fullsized":"2024-03-23T16:01:26.385328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.459675+0000","last_change":"2024-03-23T15:47:27.454131+0000","last_active":"2024-03-23T16:01:26.459675+0000","last_peered":"2024-03-23T16:01:26.459675+0000","last_clean":"2024-03-23T16:01:26.459675+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T16:01:26.459675+0000","last_undegraded":"2024-03-23T16:01:26.459675+0000","last_fullsized":"2024-03-23T16:01:26.459675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:06.449312+0000","last_change":"2024-03-23T15:47:26.150343+0000","last_active":"2024-03-23T15:59:06.449312+0000","last_peered":"2024-03-23T15:59:06.449312+0000","last_clean":"2024-03-23T15:59:06.449312+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T15:59:06.449312+0000","last_undegraded":"2024-03-23T15:59:06.449312+0000","last_fullsized":"2024-03-23T15:59:06.449312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.431905+0000","last_change":"2024-03-23T15:47:27.454743+0000","last_active":"2024-03-23T16:01:26.431905+0000","last_peered":"2024-03-23T16:01:26.431905+0000","last_clean":"2024-03-23T16:01:26.431905+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T16:01:26.431905+0000","last_undegraded":"2024-03-23T16:01:26.431905+0000","last_fullsized":"2024-03-23T16:01:26.431905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.383007+0000","last_change":"2024-03-23T15:47:26.155114+0000","last_active":"2024-03-23T16:00:56.383007+0000","last_peered":"2024-03-23T16:00:56.383007+0000","last_clean":"2024-03-23T16:00:56.383007+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T16:00:56.383007+0000","last_undegraded":"2024-03-23T16:00:56.383007+0000","last_fullsized":"2024-03-23T16:00:56.383007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.436600+0000","last_change":"2024-03-23T15:47:27.449463+0000","last_active":"2024-03-23T16:00:56.436600+0000","last_peered":"2024-03-23T16:00:56.436600+0000","last_clean":"2024-03-23T16:00:56.436600+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T16:00:56.436600+0000","last_undegraded":"2024-03-23T16:00:56.436600+0000","last_fullsized":"2024-03-23T16:00:56.436600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.446943+0000","last_change":"2024-03-23T15:47:26.151970+0000","last_active":"2024-03-23T15:58:41.446943+0000","last_peered":"2024-03-23T15:58:41.446943+0000","last_clean":"2024-03-23T15:58:41.446943+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T15:58:41.446943+0000","last_undegraded":"2024-03-23T15:58:41.446943+0000","last_fullsized":"2024-03-23T15:58:41.446943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.382998+0000","last_change":"2024-03-23T15:47:27.451409+0000","last_active":"2024-03-23T16:00:56.382998+0000","last_peered":"2024-03-23T16:00:56.382998+0000","last_clean":"2024-03-23T16:00:56.382998+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T16:00:56.382998+0000","last_undegraded":"2024-03-23T16:00:56.382998+0000","last_fullsized":"2024-03-23T16:00:56.382998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.303343+0000","last_change":"2024-03-23T15:47:26.147301+0000","last_active":"2024-03-23T15:58:41.303343+0000","last_peered":"2024-03-23T15:58:41.303343+0000","last_clean":"2024-03-23T15:58:41.303343+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T15:58:41.303343+0000","last_undegraded":"2024-03-23T15:58:41.303343+0000","last_fullsized":"2024-03-23T15:58:41.303343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:34:27.356751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.382983+0000","last_change":"2024-03-23T15:47:27.449926+0000","last_active":"2024-03-23T16:00:56.382983+0000","last_peered":"2024-03-23T16:00:56.382983+0000","last_clean":"2024-03-23T16:00:56.382983+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T16:00:56.382983+0000","last_undegraded":"2024-03-23T16:00:56.382983+0000","last_fullsized":"2024-03-23T16:00:56.382983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.364338+0000","last_change":"2024-03-23T15:47:26.152843+0000","last_active":"2024-03-23T15:58:41.364338+0000","last_peered":"2024-03-23T15:58:41.364338+0000","last_clean":"2024-03-23T15:58:41.364338+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T15:58:41.364338+0000","last_undegraded":"2024-03-23T15:58:41.364338+0000","last_fullsized":"2024-03-23T15:58:41.364338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:40:50.233814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.429354+0000","last_change":"2024-03-23T15:47:27.457334+0000","last_active":"2024-03-23T16:00:56.429354+0000","last_peered":"2024-03-23T16:00:56.429354+0000","last_clean":"2024-03-23T16:00:56.429354+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T16:00:56.429354+0000","last_undegraded":"2024-03-23T16:00:56.429354+0000","last_fullsized":"2024-03-23T16:00:56.429354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'119","reported_seq":195,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:34.761995+0000","last_change":"2024-03-23T16:01:34.762020+0000","last_active":"2024-03-23T16:01:34.761995+0000","last_peered":"2024-03-23T16:01:34.761995+0000","last_clean":"2024-03-23T16:01:34.761995+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T16:01:34.761995+0000","last_undegraded":"2024-03-23T16:01:34.761995+0000","last_fullsized":"2024-03-23T16:01:34.761995+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'119","last_scrub_stamp":"2024-03-23T16:01:34.761938+0000","last_deep_scrub":"24'119","last_deep_scrub_stamp":"2024-03-23T16:01:34.761938+0000","last_clean_scrub_stamp":"2024-03-23T16:01:34.761938+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:35.812801+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.424084+0000","last_change":"2024-03-23T15:47:26.149120+0000","last_active":"2024-03-23T15:59:56.424084+0000","last_peered":"2024-03-23T15:59:56.424084+0000","last_clean":"2024-03-23T15:59:56.424084+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T15:59:56.424084+0000","last_undegraded":"2024-03-23T15:59:56.424084+0000","last_fullsized":"2024-03-23T15:59:56.424084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442763+0000","last_change":"2024-03-23T15:47:27.451262+0000","last_active":"2024-03-23T15:57:51.442763+0000","last_peered":"2024-03-23T15:57:51.442763+0000","last_clean":"2024-03-23T15:57:51.442763+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T15:57:51.442763+0000","last_undegraded":"2024-03-23T15:57:51.442763+0000","last_fullsized":"2024-03-23T15:57:51.442763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:05.828110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:26.431886+0000","last_change":"2024-03-23T15:47:26.147520+0000","last_active":"2024-03-23T16:01:26.431886+0000","last_peered":"2024-03-23T16:01:26.431886+0000","last_clean":"2024-03-23T16:01:26.431886+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T16:01:26.431886+0000","last_undegraded":"2024-03-23T16:01:26.431886+0000","last_fullsized":"2024-03-23T16:01:26.431886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.359976+0000","last_change":"2024-03-23T15:47:27.449440+0000","last_active":"2024-03-23T15:57:51.359976+0000","last_peered":"2024-03-23T15:57:51.359976+0000","last_clean":"2024-03-23T15:57:51.359976+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T15:57:51.359976+0000","last_undegraded":"2024-03-23T15:57:51.359976+0000","last_fullsized":"2024-03-23T15:57:51.359976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:53:41.280475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.377759+0000","last_change":"2024-03-23T15:47:26.146892+0000","last_active":"2024-03-23T15:57:51.377759+0000","last_peered":"2024-03-23T15:57:51.377759+0000","last_clean":"2024-03-23T15:57:51.377759+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T15:57:51.377759+0000","last_undegraded":"2024-03-23T15:57:51.377759+0000","last_fullsized":"2024-03-23T15:57:51.377759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307635+0000","last_change":"2024-03-23T15:47:27.451430+0000","last_active":"2024-03-23T15:59:26.307635+0000","last_peered":"2024-03-23T15:59:26.307635+0000","last_clean":"2024-03-23T15:59:26.307635+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T15:59:26.307635+0000","last_undegraded":"2024-03-23T15:59:26.307635+0000","last_fullsized":"2024-03-23T15:59:26.307635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:08:39.568420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.424064+0000","last_change":"2024-03-23T15:47:26.150110+0000","last_active":"2024-03-23T15:59:56.424064+0000","last_peered":"2024-03-23T15:59:56.424064+0000","last_clean":"2024-03-23T15:59:56.424064+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T15:59:56.424064+0000","last_undegraded":"2024-03-23T15:59:56.424064+0000","last_fullsized":"2024-03-23T15:59:56.424064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.377750+0000","last_change":"2024-03-23T15:47:27.449125+0000","last_active":"2024-03-23T15:59:56.377750+0000","last_peered":"2024-03-23T15:59:56.377750+0000","last_clean":"2024-03-23T15:59:56.377750+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T15:59:56.377750+0000","last_undegraded":"2024-03-23T15:59:56.377750+0000","last_fullsized":"2024-03-23T15:59:56.377750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298792+0000","last_change":"2024-03-23T15:47:26.150159+0000","last_active":"2024-03-23T15:57:51.298792+0000","last_peered":"2024-03-23T15:57:51.298792+0000","last_clean":"2024-03-23T15:57:51.298792+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T15:57:51.298792+0000","last_undegraded":"2024-03-23T15:57:51.298792+0000","last_fullsized":"2024-03-23T15:57:51.298792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.432140+0000","last_change":"2024-03-23T15:47:27.443179+0000","last_active":"2024-03-23T15:59:56.432140+0000","last_peered":"2024-03-23T15:59:56.432140+0000","last_clean":"2024-03-23T15:59:56.432140+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T15:59:56.432140+0000","last_undegraded":"2024-03-23T15:59:56.432140+0000","last_fullsized":"2024-03-23T15:59:56.432140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.424119+0000","last_change":"2024-03-23T15:47:26.147867+0000","last_active":"2024-03-23T15:59:56.424119+0000","last_peered":"2024-03-23T15:59:56.424119+0000","last_clean":"2024-03-23T15:59:56.424119+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T15:59:56.424119+0000","last_undegraded":"2024-03-23T15:59:56.424119+0000","last_fullsized":"2024-03-23T15:59:56.424119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.360026+0000","last_change":"2024-03-23T15:47:27.383134+0000","last_active":"2024-03-23T15:57:51.360026+0000","last_peered":"2024-03-23T15:57:51.360026+0000","last_clean":"2024-03-23T15:57:51.360026+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T15:57:51.360026+0000","last_undegraded":"2024-03-23T15:57:51.360026+0000","last_fullsized":"2024-03-23T15:57:51.360026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:57:50.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444935+0000","last_change":"2024-03-23T15:47:26.153924+0000","last_active":"2024-03-23T15:58:16.444935+0000","last_peered":"2024-03-23T15:58:16.444935+0000","last_clean":"2024-03-23T15:58:16.444935+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T15:58:16.444935+0000","last_undegraded":"2024-03-23T15:58:16.444935+0000","last_fullsized":"2024-03-23T15:58:16.444935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:45.568099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444947+0000","last_change":"2024-03-23T15:47:27.438680+0000","last_active":"2024-03-23T15:58:16.444947+0000","last_peered":"2024-03-23T15:58:16.444947+0000","last_clean":"2024-03-23T15:58:16.444947+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T15:58:16.444947+0000","last_undegraded":"2024-03-23T15:58:16.444947+0000","last_fullsized":"2024-03-23T15:58:16.444947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:58:36.293329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:26.426610+0000","last_change":"2024-03-23T15:47:26.151276+0000","last_active":"2024-03-23T16:00:26.426610+0000","last_peered":"2024-03-23T16:00:26.426610+0000","last_clean":"2024-03-23T16:00:26.426610+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T16:00:26.426610+0000","last_undegraded":"2024-03-23T16:00:26.426610+0000","last_fullsized":"2024-03-23T16:00:26.426610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444921+0000","last_change":"2024-03-23T15:47:27.448525+0000","last_active":"2024-03-23T15:58:16.444921+0000","last_peered":"2024-03-23T15:58:16.444921+0000","last_clean":"2024-03-23T15:58:16.444921+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T15:58:16.444921+0000","last_undegraded":"2024-03-23T15:58:16.444921+0000","last_fullsized":"2024-03-23T15:58:16.444921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:51.073325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:26.380191+0000","last_change":"2024-03-23T15:47:26.153606+0000","last_active":"2024-03-23T16:00:26.380191+0000","last_peered":"2024-03-23T16:00:26.380191+0000","last_clean":"2024-03-23T16:00:26.380191+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T16:00:26.380191+0000","last_undegraded":"2024-03-23T16:00:26.380191+0000","last_fullsized":"2024-03-23T16:00:26.380191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:26.426572+0000","last_change":"2024-03-23T15:47:27.450648+0000","last_active":"2024-03-23T16:00:26.426572+0000","last_peered":"2024-03-23T16:00:26.426572+0000","last_clean":"2024-03-23T16:00:26.426572+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T16:00:26.426572+0000","last_undegraded":"2024-03-23T16:00:26.426572+0000","last_fullsized":"2024-03-23T16:00:26.426572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444897+0000","last_change":"2024-03-23T15:47:26.154331+0000","last_active":"2024-03-23T15:58:16.444897+0000","last_peered":"2024-03-23T15:58:16.444897+0000","last_clean":"2024-03-23T15:58:16.444897+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T15:58:16.444897+0000","last_undegraded":"2024-03-23T15:58:16.444897+0000","last_fullsized":"2024-03-23T15:58:16.444897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:17:59.661034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.444865+0000","last_change":"2024-03-23T15:47:27.448480+0000","last_active":"2024-03-23T15:58:16.444865+0000","last_peered":"2024-03-23T15:58:16.444865+0000","last_clean":"2024-03-23T15:58:16.444865+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T15:58:16.444865+0000","last_undegraded":"2024-03-23T15:58:16.444865+0000","last_fullsized":"2024-03-23T15:58:16.444865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:07:54.600862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:16.362160+0000","last_change":"2024-03-23T15:47:26.151260+0000","last_active":"2024-03-23T15:58:16.362160+0000","last_peered":"2024-03-23T15:58:16.362160+0000","last_clean":"2024-03-23T15:58:16.362160+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T15:58:16.362160+0000","last_undegraded":"2024-03-23T15:58:16.362160+0000","last_fullsized":"2024-03-23T15:58:16.362160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:05:07.072479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:58:41.381785+0000","last_change":"2024-03-23T15:47:27.448241+0000","last_active":"2024-03-23T15:58:41.381785+0000","last_peered":"2024-03-23T15:58:41.381785+0000","last_clean":"2024-03-23T15:58:41.381785+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T15:58:41.381785+0000","last_undegraded":"2024-03-23T15:58:41.381785+0000","last_fullsized":"2024-03-23T15:58:41.381785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:01.549461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:56.429330+0000","last_change":"2024-03-23T15:47:26.148721+0000","last_active":"2024-03-23T16:00:56.429330+0000","last_peered":"2024-03-23T16:00:56.429330+0000","last_clean":"2024-03-23T16:00:56.429330+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T16:00:56.429330+0000","last_undegraded":"2024-03-23T16:00:56.429330+0000","last_fullsized":"2024-03-23T16:00:56.429330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:00:26.426582+0000","last_change":"2024-03-23T15:47:27.456721+0000","last_active":"2024-03-23T16:00:26.426582+0000","last_peered":"2024-03-23T16:00:26.426582+0000","last_clean":"2024-03-23T16:00:26.426582+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T16:00:26.426582+0000","last_undegraded":"2024-03-23T16:00:26.426582+0000","last_fullsized":"2024-03-23T16:00:26.426582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.424101+0000","last_change":"2024-03-23T15:47:27.453197+0000","last_active":"2024-03-23T15:59:56.424101+0000","last_peered":"2024-03-23T15:59:56.424101+0000","last_clean":"2024-03-23T15:59:56.424101+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T15:59:56.424101+0000","last_undegraded":"2024-03-23T15:59:56.424101+0000","last_fullsized":"2024-03-23T15:59:56.424101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"24'8","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:35.301725+0000","last_change":"2024-03-23T16:01:35.301779+0000","last_active":"2024-03-23T16:01:35.301725+0000","last_peered":"2024-03-23T16:01:35.301725+0000","last_clean":"2024-03-23T16:01:35.301725+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T16:01:35.301725+0000","last_undegraded":"2024-03-23T16:01:35.301725+0000","last_fullsized":"2024-03-23T16:01:35.301725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-23T16:01:35.301625+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-23T16:01:35.301625+0000","last_clean_scrub_stamp":"2024-03-23T16:01:35.301625+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:25:55.415003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:56.452341+0000","last_change":"2024-03-23T15:47:27.455425+0000","last_active":"2024-03-23T15:59:56.452341+0000","last_peered":"2024-03-23T15:59:56.452341+0000","last_clean":"2024-03-23T15:59:56.452341+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T15:59:56.452341+0000","last_undegraded":"2024-03-23T15:59:56.452341+0000","last_fullsized":"2024-03-23T15:59:56.452341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"24'19","reported_seq":53,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:34.278181+0000","last_change":"2024-03-23T16:01:34.278216+0000","last_active":"2024-03-23T16:01:34.278181+0000","last_peered":"2024-03-23T16:01:34.278181+0000","last_clean":"2024-03-23T16:01:34.278181+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T16:01:34.278181+0000","last_undegraded":"2024-03-23T16:01:34.278181+0000","last_fullsized":"2024-03-23T16:01:34.278181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'19","last_scrub_stamp":"2024-03-23T16:01:34.278119+0000","last_deep_scrub":"24'19","last_deep_scrub_stamp":"2024-03-23T16:01:34.278119+0000","last_clean_scrub_stamp":"2024-03-23T16:01:34.278119+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:35.662485+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.298841+0000","last_change":"2024-03-23T15:47:27.450873+0000","last_active":"2024-03-23T15:57:51.298841+0000","last_peered":"2024-03-23T15:57:51.298841+0000","last_clean":"2024-03-23T15:57:51.298841+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T15:57:51.298841+0000","last_undegraded":"2024-03-23T15:57:51.298841+0000","last_fullsized":"2024-03-23T15:57:51.298841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:34:07.446471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307684+0000","last_change":"2024-03-23T15:47:26.144763+0000","last_active":"2024-03-23T15:59:26.307684+0000","last_peered":"2024-03-23T15:59:26.307684+0000","last_clean":"2024-03-23T15:59:26.307684+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T15:59:26.307684+0000","last_undegraded":"2024-03-23T15:59:26.307684+0000","last_fullsized":"2024-03-23T15:59:26.307684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:49:03.828614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:59:26.307673+0000","last_change":"2024-03-23T15:47:27.450426+0000","last_active":"2024-03-23T15:59:26.307673+0000","last_peered":"2024-03-23T15:59:26.307673+0000","last_clean":"2024-03-23T15:59:26.307673+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T15:59:26.307673+0000","last_undegraded":"2024-03-23T15:59:26.307673+0000","last_fullsized":"2024-03-23T15:59:26.307673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:26.008028+0000","last_clean_scrub_stamp":"2024-03-23T15:47:26.008028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:51.565374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T15:57:51.442806+0000","last_change":"2024-03-23T15:47:26.155041+0000","last_active":"2024-03-23T15:57:51.442806+0000","last_peered":"2024-03-23T15:57:51.442806+0000","last_clean":"2024-03-23T15:57:51.442806+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T15:57:51.442806+0000","last_undegraded":"2024-03-23T15:57:51.442806+0000","last_fullsized":"2024-03-23T15:57:51.442806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T15:47:24.873218+0000","last_clean_scrub_stamp":"2024-03-23T15:47:24.873218+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:43:05.728387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575025,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":290816,"data_stored":95068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575024,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":96394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575024,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":344064,"data_stored":685208,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575025,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":311296,"data_stored":97464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575025,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575025,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":95694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":685796,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575025,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":96556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.820 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.821 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.822 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.823 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.824 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.825 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.826 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.827 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.828 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.829 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.829 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.829 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-23T15:47:26.008028+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=26, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.829 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-03-23T15:47:24.873218+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=15, tm_min=47, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=16, tm_min=1, tm_sec=31, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T16:01:39.829 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T16:01:59.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T16:01:59.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.932+0000 7ffa034eb640 1 Processor -- start 2024-03-23T16:01:59.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.932+0000 7ffa034eb640 1 -- start start 2024-03-23T16:01:59.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.933+0000 7ffa034eb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff9fc06c410 0x7ff9fc15d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:59.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.933+0000 7ffa034eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:59.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.933+0000 7ffa034eb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9fc06c830 con 0x7ff9fc06c030 2024-03-23T16:01:59.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.933+0000 7ffa034eb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff9fc0cc370 con 0x7ff9fc06c410 2024-03-23T16:01:59.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.933+0000 7ffa034eb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc06cae0 con 0x7ff9fc15de40 2024-03-23T16:01:59.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.933+0000 7ffa01a88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:59.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.934+0000 7ffa00a86640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff9fc06c410 0x7ff9fc15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:59.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.934+0000 7ffa01a88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc15e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:39514/0 (socket says 172.21.15.129:39514) 2024-03-23T16:01:59.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.934+0000 7ffa01a88640 1 -- 172.21.15.129:0/29834681 learned_addr learned my addr 172.21.15.129:0/29834681 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:59.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.937+0000 7ffa01a88640 1 -- 172.21.15.129:0/29834681 >> v2:172.21.15.157:3300/0 conn(0x7ff9fc06c410 msgr2=0x7ff9fc15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:59.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.937+0000 7ffa01a88640 1 --2- 172.21.15.129:0/29834681 >> v2:172.21.15.157:3300/0 conn(0x7ff9fc06c410 0x7ff9fc15d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:59.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.937+0000 7ffa01a88640 1 -- 172.21.15.129:0/29834681 >> v1:172.21.15.157:6789/0 conn(0x7ff9fc06c030 legacy=0x7ff9fc0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T16:01:59.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.937+0000 7ffa01a88640 1 -- 172.21.15.129:0/29834681 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc0cabc0 con 0x7ff9fc15de40 2024-03-23T16:01:59.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.940+0000 7ffa01a88640 1 --2- 172.21.15.129:0/29834681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc15e2b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009e50 tx=0x7ff9f402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45a2f3e203e8735b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:59.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.945+0000 7ff9f3fff640 1 -- 172.21.15.129:0/29834681 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f402ca70 con 0x7ff9fc15de40 2024-03-23T16:01:59.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.945+0000 7ff9f3fff640 1 -- 172.21.15.129:0/29834681 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f402cbd0 con 0x7ff9fc15de40 2024-03-23T16:01:59.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.945+0000 7ff9f3fff640 1 -- 172.21.15.129:0/29834681 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40343f0 con 0x7ff9fc15de40 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 -- 172.21.15.129:0/29834681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 msgr2=0x7ff9fc15e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 --2- 172.21.15.129:0/29834681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc15e2b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009e50 tx=0x7ff9f402bed0 comp rx=0 tx=0).stop 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 -- 172.21.15.129:0/29834681 shutdown_connections 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 --2- 172.21.15.129:0/29834681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc15e2b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 --2- 172.21.15.129:0/29834681 >> v2:172.21.15.157:3300/0 conn(0x7ff9fc06c410 0x7ff9fc15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 -- 172.21.15.129:0/29834681 >> 172.21.15.129:0/29834681 conn(0x7ff9fc0cfb20 msgr2=0x7ff9fc0cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 -- 172.21.15.129:0/29834681 shutdown_connections 2024-03-23T16:01:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.946+0000 7ffa034eb640 1 -- 172.21.15.129:0/29834681 wait complete. 2024-03-23T16:01:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.947+0000 7ffa034eb640 1 Processor -- start 2024-03-23T16:01:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.947+0000 7ffa034eb640 1 -- start start 2024-03-23T16:01:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.948+0000 7ffa034eb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9fc06c030 0x7ff9fc156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.948+0000 7ffa034eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 0x7ff9fc155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.948+0000 7ffa034eb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc1f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.948+0000 7ffa034eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc157420 con 0x7ff9fc06c410 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.948+0000 7ffa034eb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc154d00 con 0x7ff9fc06c030 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.948+0000 7ffa034eb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc154fc0 con 0x7ff9fc15de40 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.949+0000 7ffa00a86640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 0x7ff9fc155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.949+0000 7ffa01287640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9fc06c030 0x7ff9fc156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.949+0000 7ffa01a88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc1f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.949+0000 7ffa00a86640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 0x7ff9fc155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35268/0 (socket says 172.21.15.129:35268) 2024-03-23T16:01:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.949+0000 7ffa00a86640 1 -- 172.21.15.129:0/813552514 learned_addr learned my addr 172.21.15.129:0/813552514 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:01:59.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.952+0000 7ffa00a86640 1 -- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 msgr2=0x7ff9fc1f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:59.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.952+0000 7ffa00a86640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc1f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:59.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.952+0000 7ffa00a86640 1 -- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9fc06c030 msgr2=0x7ff9fc156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:01:59.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.952+0000 7ffa00a86640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9fc06c030 0x7ff9fc156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:01:59.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.952+0000 7ffa00a86640 1 -- 172.21.15.129:0/813552514 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f4009b30 con 0x7ff9fc06c410 2024-03-23T16:01:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.955+0000 7ffa00a86640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 0x7ff9fc155cd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00e9b0 tx=0x7ff9ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.958+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec009670 con 0x7ff9fc06c410 2024-03-23T16:01:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.958+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec00c9a0 con 0x7ff9fc06c410 2024-03-23T16:01:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.958+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec00cd50 con 0x7ff9fc06c410 2024-03-23T16:01:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.958+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc1f3da0 con 0x7ff9fc06c410 2024-03-23T16:01:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.958+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc1514c0 con 0x7ff9fc06c410 2024-03-23T16:01:59.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.960+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc0b50b0 con 0x7ff9fc06c410 2024-03-23T16:01:59.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.979+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff9ec013470 con 0x7ff9fc06c410 2024-03-23T16:01:59.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.979+0000 7ff9f1ffb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff9e006b860 0x7ff9e006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:01:59.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.980+0000 7ffa01287640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff9e006b860 0x7ff9e006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:01:59.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.980+0000 7ffa01287640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff9e006b860 0x7ff9e006dd20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8004f60 tx=0x7ff9f8007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:01:59.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:01:59.983+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7ff9ec014e40 con 0x7ff9fc06c410 2024-03-23T16:02:00.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.253+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff9ec055660 con 0x7ff9fc06c410 2024-03-23T16:02:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.444+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 --> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9fc0cb460 con 0x7ff9e006b860 2024-03-23T16:02:00.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.450+0000 7ff9f1ffb640 1 -- 172.21.15.129:0/813552514 <== mgr.4101 v2:172.21.15.157:6800/2961890009 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252135 (secure 0 0 0) 0x7ff9f8003940 con 0x7ff9e006b860 2024-03-23T16:02:00.455 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-23T16:02:00.456 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-23T16:02:00.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff9e006b860 msgr2=0x7ff9e006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff9e006b860 0x7ff9e006dd20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8004f60 tx=0x7ff9f8007e30 comp rx=0 tx=0).stop 2024-03-23T16:02:00.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 msgr2=0x7ff9fc155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 0x7ff9fc155cd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00e9b0 tx=0x7ff9ec00ee70 comp rx=0 tx=0).stop 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 shutdown_connections 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9fc15de40 0x7ff9fc1f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9fc06c410 0x7ff9fc155cd0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7ff9e006b860 0x7ff9e006dd20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 --2- 172.21.15.129:0/813552514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9fc06c030 0x7ff9fc156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 >> 172.21.15.129:0/813552514 conn(0x7ff9fc0cfb20 msgr2=0x7ff9fc0c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 shutdown_connections 2024-03-23T16:02:00.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.454+0000 7ffa034eb640 1 -- 172.21.15.129:0/813552514 wait complete. 2024-03-23T16:02:00.467 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-03-23T16:01:58.589697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":125,"num_read_kb":108,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235136,"kb_used_data":2368,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739584,"statfs":{"total":773094113280,"available":772853334016,"internally_reserved":0,"allocated":2424832,"data_stored":1948916,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002528"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:49.463692+0000","last_change":"2024-03-23T16:01:49.463710+0000","last_active":"2024-03-23T16:01:49.463692+0000","last_peered":"2024-03-23T16:01:49.463692+0000","last_clean":"2024-03-23T16:01:49.463692+0000","last_became_active":"2024-03-23T15:47:27.453234+0000","last_became_peered":"2024-03-23T15:47:27.453234+0000","last_unstale":"2024-03-23T16:01:49.463692+0000","last_undegraded":"2024-03-23T16:01:49.463692+0000","last_fullsized":"2024-03-23T16:01:49.463692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:49.463620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:49.463620+0000","last_clean_scrub_stamp":"2024-03-23T16:01:49.463620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:59:48.943119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.334233+0000","last_change":"2024-03-23T16:01:42.334270+0000","last_active":"2024-03-23T16:01:42.334233+0000","last_peered":"2024-03-23T16:01:42.334233+0000","last_clean":"2024-03-23T16:01:42.334233+0000","last_became_active":"2024-03-23T15:47:27.382661+0000","last_became_peered":"2024-03-23T15:47:27.382661+0000","last_unstale":"2024-03-23T16:01:42.334233+0000","last_undegraded":"2024-03-23T16:01:42.334233+0000","last_fullsized":"2024-03-23T16:01:42.334233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:42.334128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:42.334128+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.334128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:05.356369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:46.281936+0000","last_change":"2024-03-23T16:01:46.281970+0000","last_active":"2024-03-23T16:01:46.281936+0000","last_peered":"2024-03-23T16:01:46.281936+0000","last_clean":"2024-03-23T16:01:46.281936+0000","last_became_active":"2024-03-23T15:47:27.438716+0000","last_became_peered":"2024-03-23T15:47:27.438716+0000","last_unstale":"2024-03-23T16:01:46.281936+0000","last_undegraded":"2024-03-23T16:01:46.281936+0000","last_fullsized":"2024-03-23T16:01:46.281936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:46.281836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:46.281836+0000","last_clean_scrub_stamp":"2024-03-23T16:01:46.281836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:21:29.017425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.021618+0000","last_change":"2024-03-23T16:01:41.021638+0000","last_active":"2024-03-23T16:01:41.021618+0000","last_peered":"2024-03-23T16:01:41.021618+0000","last_clean":"2024-03-23T16:01:41.021618+0000","last_became_active":"2024-03-23T15:47:27.449473+0000","last_became_peered":"2024-03-23T15:47:27.449473+0000","last_unstale":"2024-03-23T16:01:41.021618+0000","last_undegraded":"2024-03-23T16:01:41.021618+0000","last_fullsized":"2024-03-23T16:01:41.021618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:41.021552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:41.021552+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.021552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:24:07.280364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.416730+0000","last_change":"2024-03-23T16:01:41.416764+0000","last_active":"2024-03-23T16:01:41.416730+0000","last_peered":"2024-03-23T16:01:41.416730+0000","last_clean":"2024-03-23T16:01:41.416730+0000","last_became_active":"2024-03-23T15:47:27.382702+0000","last_became_peered":"2024-03-23T15:47:27.382702+0000","last_unstale":"2024-03-23T16:01:41.416730+0000","last_undegraded":"2024-03-23T16:01:41.416730+0000","last_fullsized":"2024-03-23T16:01:41.416730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:41.416623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:41.416623+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.416623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:44.671795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.035432+0000","last_change":"2024-03-23T16:01:39.035452+0000","last_active":"2024-03-23T16:01:39.035432+0000","last_peered":"2024-03-23T16:01:39.035432+0000","last_clean":"2024-03-23T16:01:39.035432+0000","last_became_active":"2024-03-23T15:47:27.442909+0000","last_became_peered":"2024-03-23T15:47:27.442909+0000","last_unstale":"2024-03-23T16:01:39.035432+0000","last_undegraded":"2024-03-23T16:01:39.035432+0000","last_fullsized":"2024-03-23T16:01:39.035432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:39.035367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:39.035367+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.035367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:10:47.684209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.438137+0000","last_change":"2024-03-23T16:01:42.438172+0000","last_active":"2024-03-23T16:01:42.438137+0000","last_peered":"2024-03-23T16:01:42.438137+0000","last_clean":"2024-03-23T16:01:42.438137+0000","last_became_active":"2024-03-23T15:47:27.438507+0000","last_became_peered":"2024-03-23T15:47:27.438507+0000","last_unstale":"2024-03-23T16:01:42.438137+0000","last_undegraded":"2024-03-23T16:01:42.438137+0000","last_fullsized":"2024-03-23T16:01:42.438137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:42.438031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:42.438031+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.438031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:12.260874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:44.336445+0000","last_change":"2024-03-23T16:01:44.336481+0000","last_active":"2024-03-23T16:01:44.336445+0000","last_peered":"2024-03-23T16:01:44.336445+0000","last_clean":"2024-03-23T16:01:44.336445+0000","last_became_active":"2024-03-23T15:47:27.382964+0000","last_became_peered":"2024-03-23T15:47:27.382964+0000","last_unstale":"2024-03-23T16:01:44.336445+0000","last_undegraded":"2024-03-23T16:01:44.336445+0000","last_fullsized":"2024-03-23T16:01:44.336445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:44.336330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:44.336330+0000","last_clean_scrub_stamp":"2024-03-23T16:01:44.336330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:47.591502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.656707+0000","last_change":"2024-03-23T16:01:40.656707+0000","last_active":"2024-03-23T16:01:40.656707+0000","last_peered":"2024-03-23T16:01:40.656707+0000","last_clean":"2024-03-23T16:01:40.656707+0000","last_became_active":"2024-03-23T15:47:27.456171+0000","last_became_peered":"2024-03-23T15:47:27.456171+0000","last_unstale":"2024-03-23T16:01:40.656707+0000","last_undegraded":"2024-03-23T16:01:40.656707+0000","last_fullsized":"2024-03-23T16:01:40.656707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:40.656573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:40.656573+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.656573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:24:06.915385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:47.504370+0000","last_change":"2024-03-23T16:01:47.504370+0000","last_active":"2024-03-23T16:01:47.504370+0000","last_peered":"2024-03-23T16:01:47.504370+0000","last_clean":"2024-03-23T16:01:47.504370+0000","last_became_active":"2024-03-23T15:47:27.450705+0000","last_became_peered":"2024-03-23T15:47:27.450705+0000","last_unstale":"2024-03-23T16:01:47.504370+0000","last_undegraded":"2024-03-23T16:01:47.504370+0000","last_fullsized":"2024-03-23T16:01:47.504370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:47.504256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:47.504256+0000","last_clean_scrub_stamp":"2024-03-23T16:01:47.504256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:03:27.430328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:45.395492+0000","last_change":"2024-03-23T16:01:45.395535+0000","last_active":"2024-03-23T16:01:45.395492+0000","last_peered":"2024-03-23T16:01:45.395492+0000","last_clean":"2024-03-23T16:01:45.395492+0000","last_became_active":"2024-03-23T15:48:04.931241+0000","last_became_peered":"2024-03-23T15:48:04.931241+0000","last_unstale":"2024-03-23T16:01:45.395492+0000","last_undegraded":"2024-03-23T16:01:45.395492+0000","last_fullsized":"2024-03-23T16:01:45.395492+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:45.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:45.395375+0000","last_clean_scrub_stamp":"2024-03-23T16:01:45.395375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:12:17.693942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:48.497528+0000","last_change":"2024-03-23T16:01:48.497528+0000","last_active":"2024-03-23T16:01:48.497528+0000","last_peered":"2024-03-23T16:01:48.497528+0000","last_clean":"2024-03-23T16:01:48.497528+0000","last_became_active":"2024-03-23T15:47:27.452380+0000","last_became_peered":"2024-03-23T15:47:27.452380+0000","last_unstale":"2024-03-23T16:01:48.497528+0000","last_undegraded":"2024-03-23T16:01:48.497528+0000","last_fullsized":"2024-03-23T16:01:48.497528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:48.497419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:48.497419+0000","last_clean_scrub_stamp":"2024-03-23T16:01:48.497419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:27.862880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.876516+0000","last_change":"2024-03-23T16:01:42.876516+0000","last_active":"2024-03-23T16:01:42.876516+0000","last_peered":"2024-03-23T16:01:42.876516+0000","last_clean":"2024-03-23T16:01:42.876516+0000","last_became_active":"2024-03-23T15:47:27.355579+0000","last_became_peered":"2024-03-23T15:47:27.355579+0000","last_unstale":"2024-03-23T16:01:42.876516+0000","last_undegraded":"2024-03-23T16:01:42.876516+0000","last_fullsized":"2024-03-23T16:01:42.876516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:42.876423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:42.876423+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.876423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:45:02.591195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:45.912691+0000","last_change":"2024-03-23T16:01:45.912728+0000","last_active":"2024-03-23T16:01:45.912691+0000","last_peered":"2024-03-23T16:01:45.912691+0000","last_clean":"2024-03-23T16:01:45.912691+0000","last_became_active":"2024-03-23T15:47:27.448800+0000","last_became_peered":"2024-03-23T15:47:27.448800+0000","last_unstale":"2024-03-23T16:01:45.912691+0000","last_undegraded":"2024-03-23T16:01:45.912691+0000","last_fullsized":"2024-03-23T16:01:45.912691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:45.912584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:45.912584+0000","last_clean_scrub_stamp":"2024-03-23T16:01:45.912584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:08.746519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.346013+0000","last_change":"2024-03-23T16:01:43.346033+0000","last_active":"2024-03-23T16:01:43.346013+0000","last_peered":"2024-03-23T16:01:43.346013+0000","last_clean":"2024-03-23T16:01:43.346013+0000","last_became_active":"2024-03-23T15:47:27.383420+0000","last_became_peered":"2024-03-23T15:47:27.383420+0000","last_unstale":"2024-03-23T16:01:43.346013+0000","last_undegraded":"2024-03-23T16:01:43.346013+0000","last_fullsized":"2024-03-23T16:01:43.346013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:43.345947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:43.345947+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.345947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:21:15.932468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:54.049622+0000","last_change":"2024-03-23T16:01:54.049639+0000","last_active":"2024-03-23T16:01:54.049622+0000","last_peered":"2024-03-23T16:01:54.049622+0000","last_clean":"2024-03-23T16:01:54.049622+0000","last_became_active":"2024-03-23T15:47:27.456325+0000","last_became_peered":"2024-03-23T15:47:27.456325+0000","last_unstale":"2024-03-23T16:01:54.049622+0000","last_undegraded":"2024-03-23T16:01:54.049622+0000","last_fullsized":"2024-03-23T16:01:54.049622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:54.049556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:54.049556+0000","last_clean_scrub_stamp":"2024-03-23T16:01:54.049556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:15.639326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.170661+0000","last_change":"2024-03-23T16:01:36.170703+0000","last_active":"2024-03-23T16:01:36.170661+0000","last_peered":"2024-03-23T16:01:36.170661+0000","last_clean":"2024-03-23T16:01:36.170661+0000","last_became_active":"2024-03-23T15:47:27.450999+0000","last_became_peered":"2024-03-23T15:47:27.450999+0000","last_unstale":"2024-03-23T16:01:36.170661+0000","last_undegraded":"2024-03-23T16:01:36.170661+0000","last_fullsized":"2024-03-23T16:01:36.170661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:36.170584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:36.170584+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.170584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:20:46.816752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.420027+0000","last_change":"2024-03-23T16:01:43.420062+0000","last_active":"2024-03-23T16:01:43.420027+0000","last_peered":"2024-03-23T16:01:43.420027+0000","last_clean":"2024-03-23T16:01:43.420027+0000","last_became_active":"2024-03-23T15:47:27.452645+0000","last_became_peered":"2024-03-23T15:47:27.452645+0000","last_unstale":"2024-03-23T16:01:43.420027+0000","last_undegraded":"2024-03-23T16:01:43.420027+0000","last_fullsized":"2024-03-23T16:01:43.420027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:43.419919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:43.419919+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.419919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:40:56.841669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:51.057308+0000","last_change":"2024-03-23T16:01:51.057345+0000","last_active":"2024-03-23T16:01:51.057308+0000","last_peered":"2024-03-23T16:01:51.057308+0000","last_clean":"2024-03-23T16:01:51.057308+0000","last_became_active":"2024-03-23T15:47:27.454317+0000","last_became_peered":"2024-03-23T15:47:27.454317+0000","last_unstale":"2024-03-23T16:01:51.057308+0000","last_undegraded":"2024-03-23T16:01:51.057308+0000","last_fullsized":"2024-03-23T16:01:51.057308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:51.057198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:51.057198+0000","last_clean_scrub_stamp":"2024-03-23T16:01:51.057198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:56:18.187878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:46.479183+0000","last_change":"2024-03-23T16:01:46.479202+0000","last_active":"2024-03-23T16:01:46.479183+0000","last_peered":"2024-03-23T16:01:46.479183+0000","last_clean":"2024-03-23T16:01:46.479183+0000","last_became_active":"2024-03-23T15:47:27.451783+0000","last_became_peered":"2024-03-23T15:47:27.451783+0000","last_unstale":"2024-03-23T16:01:46.479183+0000","last_undegraded":"2024-03-23T16:01:46.479183+0000","last_fullsized":"2024-03-23T16:01:46.479183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:46.479117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:46.479117+0000","last_clean_scrub_stamp":"2024-03-23T16:01:46.479117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:42:25.020843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:45.634739+0000","last_change":"2024-03-23T16:01:45.634775+0000","last_active":"2024-03-23T16:01:45.634739+0000","last_peered":"2024-03-23T16:01:45.634739+0000","last_clean":"2024-03-23T16:01:45.634739+0000","last_became_active":"2024-03-23T15:47:27.449027+0000","last_became_peered":"2024-03-23T15:47:27.449027+0000","last_unstale":"2024-03-23T16:01:45.634739+0000","last_undegraded":"2024-03-23T16:01:45.634739+0000","last_fullsized":"2024-03-23T16:01:45.634739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:45.634664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:45.634664+0000","last_clean_scrub_stamp":"2024-03-23T16:01:45.634664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:10.226388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.714725+0000","last_change":"2024-03-23T16:01:38.714761+0000","last_active":"2024-03-23T16:01:38.714725+0000","last_peered":"2024-03-23T16:01:38.714725+0000","last_clean":"2024-03-23T16:01:38.714725+0000","last_became_active":"2024-03-23T15:47:27.454680+0000","last_became_peered":"2024-03-23T15:47:27.454680+0000","last_unstale":"2024-03-23T16:01:38.714725+0000","last_undegraded":"2024-03-23T16:01:38.714725+0000","last_fullsized":"2024-03-23T16:01:38.714725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:38.714602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:38.714602+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.714602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:10:47.363443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:51.280034+0000","last_change":"2024-03-23T16:01:51.280071+0000","last_active":"2024-03-23T16:01:51.280034+0000","last_peered":"2024-03-23T16:01:51.280034+0000","last_clean":"2024-03-23T16:01:51.280034+0000","last_became_active":"2024-03-23T15:47:27.450736+0000","last_became_peered":"2024-03-23T15:47:27.450736+0000","last_unstale":"2024-03-23T16:01:51.280034+0000","last_undegraded":"2024-03-23T16:01:51.280034+0000","last_fullsized":"2024-03-23T16:01:51.280034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:51.279921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:51.279921+0000","last_clean_scrub_stamp":"2024-03-23T16:01:51.279921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:30.645383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.142378+0000","last_change":"2024-03-23T16:01:39.142398+0000","last_active":"2024-03-23T16:01:39.142378+0000","last_peered":"2024-03-23T16:01:39.142378+0000","last_clean":"2024-03-23T16:01:39.142378+0000","last_became_active":"2024-03-23T15:47:27.451221+0000","last_became_peered":"2024-03-23T15:47:27.451221+0000","last_unstale":"2024-03-23T16:01:39.142378+0000","last_undegraded":"2024-03-23T16:01:39.142378+0000","last_fullsized":"2024-03-23T16:01:39.142378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:39.142317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:39.142317+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.142317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:00:52.832656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:47.241876+0000","last_change":"2024-03-23T16:01:47.241896+0000","last_active":"2024-03-23T16:01:47.241876+0000","last_peered":"2024-03-23T16:01:47.241876+0000","last_clean":"2024-03-23T16:01:47.241876+0000","last_became_active":"2024-03-23T15:47:27.450016+0000","last_became_peered":"2024-03-23T15:47:27.450016+0000","last_unstale":"2024-03-23T16:01:47.241876+0000","last_undegraded":"2024-03-23T16:01:47.241876+0000","last_fullsized":"2024-03-23T16:01:47.241876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:47.241813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:47.241813+0000","last_clean_scrub_stamp":"2024-03-23T16:01:47.241813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:48:59.711156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.100412+0000","last_change":"2024-03-23T16:01:43.100445+0000","last_active":"2024-03-23T16:01:43.100412+0000","last_peered":"2024-03-23T16:01:43.100412+0000","last_clean":"2024-03-23T16:01:43.100412+0000","last_became_active":"2024-03-23T15:47:27.456542+0000","last_became_peered":"2024-03-23T15:47:27.456542+0000","last_unstale":"2024-03-23T16:01:43.100412+0000","last_undegraded":"2024-03-23T16:01:43.100412+0000","last_fullsized":"2024-03-23T16:01:43.100412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:43.100315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:43.100315+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.100315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:45:02.815088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:44.876070+0000","last_change":"2024-03-23T16:01:44.876103+0000","last_active":"2024-03-23T16:01:44.876070+0000","last_peered":"2024-03-23T16:01:44.876070+0000","last_clean":"2024-03-23T16:01:44.876070+0000","last_became_active":"2024-03-23T15:47:27.448159+0000","last_became_peered":"2024-03-23T15:47:27.448159+0000","last_unstale":"2024-03-23T16:01:44.876070+0000","last_undegraded":"2024-03-23T16:01:44.876070+0000","last_fullsized":"2024-03-23T16:01:44.876070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:44.875996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:44.875996+0000","last_clean_scrub_stamp":"2024-03-23T16:01:44.875996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:26:04.989373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.891601+0000","last_change":"2024-03-23T16:01:41.891619+0000","last_active":"2024-03-23T16:01:41.891601+0000","last_peered":"2024-03-23T16:01:41.891601+0000","last_clean":"2024-03-23T16:01:41.891601+0000","last_became_active":"2024-03-23T15:47:27.355512+0000","last_became_peered":"2024-03-23T15:47:27.355512+0000","last_unstale":"2024-03-23T16:01:41.891601+0000","last_undegraded":"2024-03-23T16:01:41.891601+0000","last_fullsized":"2024-03-23T16:01:41.891601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:41.891539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:41.891539+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.891539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:09:59.895879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:49.254786+0000","last_change":"2024-03-23T16:01:49.254821+0000","last_active":"2024-03-23T16:01:49.254786+0000","last_peered":"2024-03-23T16:01:49.254786+0000","last_clean":"2024-03-23T16:01:49.254786+0000","last_became_active":"2024-03-23T15:47:27.450413+0000","last_became_peered":"2024-03-23T15:47:27.450413+0000","last_unstale":"2024-03-23T16:01:49.254786+0000","last_undegraded":"2024-03-23T16:01:49.254786+0000","last_fullsized":"2024-03-23T16:01:49.254786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:49.254684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:49.254684+0000","last_clean_scrub_stamp":"2024-03-23T16:01:49.254684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:42:53.737386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:49.089796+0000","last_change":"2024-03-23T16:01:49.089814+0000","last_active":"2024-03-23T16:01:49.089796+0000","last_peered":"2024-03-23T16:01:49.089796+0000","last_clean":"2024-03-23T16:01:49.089796+0000","last_became_active":"2024-03-23T15:47:27.452155+0000","last_became_peered":"2024-03-23T15:47:27.452155+0000","last_unstale":"2024-03-23T16:01:49.089796+0000","last_undegraded":"2024-03-23T16:01:49.089796+0000","last_fullsized":"2024-03-23T16:01:49.089796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:49.089719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:49.089719+0000","last_clean_scrub_stamp":"2024-03-23T16:01:49.089719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:25:26.639617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:44.400627+0000","last_change":"2024-03-23T16:01:44.400646+0000","last_active":"2024-03-23T16:01:44.400627+0000","last_peered":"2024-03-23T16:01:44.400627+0000","last_clean":"2024-03-23T16:01:44.400627+0000","last_became_active":"2024-03-23T15:47:27.455404+0000","last_became_peered":"2024-03-23T15:47:27.455404+0000","last_unstale":"2024-03-23T16:01:44.400627+0000","last_undegraded":"2024-03-23T16:01:44.400627+0000","last_fullsized":"2024-03-23T16:01:44.400627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:44.400561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:44.400561+0000","last_clean_scrub_stamp":"2024-03-23T16:01:44.400561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:28:10.460792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:37.007617+0000","last_change":"2024-03-23T16:01:37.007636+0000","last_active":"2024-03-23T16:01:37.007617+0000","last_peered":"2024-03-23T16:01:37.007617+0000","last_clean":"2024-03-23T16:01:37.007617+0000","last_became_active":"2024-03-23T15:47:27.442851+0000","last_became_peered":"2024-03-23T15:47:27.442851+0000","last_unstale":"2024-03-23T16:01:37.007617+0000","last_undegraded":"2024-03-23T16:01:37.007617+0000","last_fullsized":"2024-03-23T16:01:37.007617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:37.007546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:37.007546+0000","last_clean_scrub_stamp":"2024-03-23T16:01:37.007546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:01:38.577029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.726124+0000","last_change":"2024-03-23T16:01:36.726163+0000","last_active":"2024-03-23T16:01:36.726124+0000","last_peered":"2024-03-23T16:01:36.726124+0000","last_clean":"2024-03-23T16:01:36.726124+0000","last_became_active":"2024-03-23T15:47:27.448074+0000","last_became_peered":"2024-03-23T15:47:27.448074+0000","last_unstale":"2024-03-23T16:01:36.726124+0000","last_undegraded":"2024-03-23T16:01:36.726124+0000","last_fullsized":"2024-03-23T16:01:36.726124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:36.726004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:36.726004+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.726004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:16:04.508461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"24'9","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.307527+0000","last_change":"2024-03-23T16:01:41.307574+0000","last_active":"2024-03-23T16:01:41.307527+0000","last_peered":"2024-03-23T16:01:41.307527+0000","last_clean":"2024-03-23T16:01:41.307527+0000","last_became_active":"2024-03-23T15:47:26.151285+0000","last_became_peered":"2024-03-23T15:47:26.151285+0000","last_unstale":"2024-03-23T16:01:41.307527+0000","last_undegraded":"2024-03-23T16:01:41.307527+0000","last_fullsized":"2024-03-23T16:01:41.307527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-23T16:01:41.307412+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-23T16:01:41.307412+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.307412+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:51:07.438588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.685513+0000","last_change":"2024-03-23T16:01:42.685548+0000","last_active":"2024-03-23T16:01:42.685513+0000","last_peered":"2024-03-23T16:01:42.685513+0000","last_clean":"2024-03-23T16:01:42.685513+0000","last_became_active":"2024-03-23T15:47:27.438568+0000","last_became_peered":"2024-03-23T15:47:27.438568+0000","last_unstale":"2024-03-23T16:01:42.685513+0000","last_undegraded":"2024-03-23T16:01:42.685513+0000","last_fullsized":"2024-03-23T16:01:42.685513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:42.685404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:42.685404+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.685404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:06:09.620146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"24'10","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:48.062813+0000","last_change":"2024-03-23T16:01:48.062856+0000","last_active":"2024-03-23T16:01:48.062813+0000","last_peered":"2024-03-23T16:01:48.062813+0000","last_clean":"2024-03-23T16:01:48.062813+0000","last_became_active":"2024-03-23T15:47:26.150981+0000","last_became_peered":"2024-03-23T15:47:26.150981+0000","last_unstale":"2024-03-23T16:01:48.062813+0000","last_undegraded":"2024-03-23T16:01:48.062813+0000","last_fullsized":"2024-03-23T16:01:48.062813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'10","last_scrub_stamp":"2024-03-23T16:01:48.062741+0000","last_deep_scrub":"24'10","last_deep_scrub_stamp":"2024-03-23T16:01:48.062741+0000","last_clean_scrub_stamp":"2024-03-23T16:01:48.062741+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:04:14.476982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"24'5","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.893403+0000","last_change":"2024-03-23T16:01:39.893403+0000","last_active":"2024-03-23T16:01:39.893403+0000","last_peered":"2024-03-23T16:01:39.893403+0000","last_clean":"2024-03-23T16:01:39.893403+0000","last_became_active":"2024-03-23T15:47:26.150909+0000","last_became_peered":"2024-03-23T15:47:26.150909+0000","last_unstale":"2024-03-23T16:01:39.893403+0000","last_undegraded":"2024-03-23T16:01:39.893403+0000","last_fullsized":"2024-03-23T16:01:39.893403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-23T16:01:39.893320+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-23T16:01:39.893320+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.893320+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:51:40.458670+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:45.320060+0000","last_change":"2024-03-23T16:01:45.320060+0000","last_active":"2024-03-23T16:01:45.320060+0000","last_peered":"2024-03-23T16:01:45.320060+0000","last_clean":"2024-03-23T16:01:45.320060+0000","last_became_active":"2024-03-23T15:47:27.384157+0000","last_became_peered":"2024-03-23T15:47:27.384157+0000","last_unstale":"2024-03-23T16:01:45.320060+0000","last_undegraded":"2024-03-23T16:01:45.320060+0000","last_fullsized":"2024-03-23T16:01:45.320060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:45.319921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:45.319921+0000","last_clean_scrub_stamp":"2024-03-23T16:01:45.319921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:21:15.142764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"24'5","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.101759+0000","last_change":"2024-03-23T16:01:43.101759+0000","last_active":"2024-03-23T16:01:43.101759+0000","last_peered":"2024-03-23T16:01:43.101759+0000","last_clean":"2024-03-23T16:01:43.101759+0000","last_became_active":"2024-03-23T15:47:26.147481+0000","last_became_peered":"2024-03-23T15:47:26.147481+0000","last_unstale":"2024-03-23T16:01:43.101759+0000","last_undegraded":"2024-03-23T16:01:43.101759+0000","last_fullsized":"2024-03-23T16:01:43.101759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-23T16:01:43.101679+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-23T16:01:43.101679+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.101679+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:59:27.869609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:46.662323+0000","last_change":"2024-03-23T16:01:46.662323+0000","last_active":"2024-03-23T16:01:46.662323+0000","last_peered":"2024-03-23T16:01:46.662323+0000","last_clean":"2024-03-23T16:01:46.662323+0000","last_became_active":"2024-03-23T15:47:27.450551+0000","last_became_peered":"2024-03-23T15:47:27.450551+0000","last_unstale":"2024-03-23T16:01:46.662323+0000","last_undegraded":"2024-03-23T16:01:46.662323+0000","last_fullsized":"2024-03-23T16:01:46.662323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:46.662173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:46.662173+0000","last_clean_scrub_stamp":"2024-03-23T16:01:46.662173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:01:00.352512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"24'7","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:35.753452+0000","last_change":"2024-03-23T16:01:35.753452+0000","last_active":"2024-03-23T16:01:35.753452+0000","last_peered":"2024-03-23T16:01:35.753452+0000","last_clean":"2024-03-23T16:01:35.753452+0000","last_became_active":"2024-03-23T15:47:26.149522+0000","last_became_peered":"2024-03-23T15:47:26.149522+0000","last_unstale":"2024-03-23T16:01:35.753452+0000","last_undegraded":"2024-03-23T16:01:35.753452+0000","last_fullsized":"2024-03-23T16:01:35.753452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-23T16:01:35.753304+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-23T16:01:35.753304+0000","last_clean_scrub_stamp":"2024-03-23T16:01:35.753304+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:06:57.213380+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.071826+0000","last_change":"2024-03-23T16:01:42.071826+0000","last_active":"2024-03-23T16:01:42.071826+0000","last_peered":"2024-03-23T16:01:42.071826+0000","last_clean":"2024-03-23T16:01:42.071826+0000","last_became_active":"2024-03-23T15:47:27.455895+0000","last_became_peered":"2024-03-23T15:47:27.455895+0000","last_unstale":"2024-03-23T16:01:42.071826+0000","last_undegraded":"2024-03-23T16:01:42.071826+0000","last_fullsized":"2024-03-23T16:01:42.071826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:42.071736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:42.071736+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.071736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:10:00.076075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"24'6","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.336158+0000","last_change":"2024-03-23T16:01:36.336158+0000","last_active":"2024-03-23T16:01:36.336158+0000","last_peered":"2024-03-23T16:01:36.336158+0000","last_clean":"2024-03-23T16:01:36.336158+0000","last_became_active":"2024-03-23T15:47:26.153003+0000","last_became_peered":"2024-03-23T15:47:26.153003+0000","last_unstale":"2024-03-23T16:01:36.336158+0000","last_undegraded":"2024-03-23T16:01:36.336158+0000","last_fullsized":"2024-03-23T16:01:36.336158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-23T16:01:36.336071+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-23T16:01:36.336071+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.336071+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:59.170006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:53.251921+0000","last_change":"2024-03-23T16:01:53.251921+0000","last_active":"2024-03-23T16:01:53.251921+0000","last_peered":"2024-03-23T16:01:53.251921+0000","last_clean":"2024-03-23T16:01:53.251921+0000","last_became_active":"2024-03-23T15:47:27.454432+0000","last_became_peered":"2024-03-23T15:47:27.454432+0000","last_unstale":"2024-03-23T16:01:53.251921+0000","last_undegraded":"2024-03-23T16:01:53.251921+0000","last_fullsized":"2024-03-23T16:01:53.251921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:53.251834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:53.251834+0000","last_clean_scrub_stamp":"2024-03-23T16:01:53.251834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:20.155543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"24'9","reported_seq":45,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.462086+0000","last_change":"2024-03-23T16:01:40.462086+0000","last_active":"2024-03-23T16:01:40.462086+0000","last_peered":"2024-03-23T16:01:40.462086+0000","last_clean":"2024-03-23T16:01:40.462086+0000","last_became_active":"2024-03-23T15:47:26.146875+0000","last_became_peered":"2024-03-23T15:47:26.146875+0000","last_unstale":"2024-03-23T16:01:40.462086+0000","last_undegraded":"2024-03-23T16:01:40.462086+0000","last_fullsized":"2024-03-23T16:01:40.462086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-23T16:01:40.461944+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-23T16:01:40.461944+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.461944+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:46.350642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:53.072851+0000","last_change":"2024-03-23T16:01:53.072871+0000","last_active":"2024-03-23T16:01:53.072851+0000","last_peered":"2024-03-23T16:01:53.072851+0000","last_clean":"2024-03-23T16:01:53.072851+0000","last_became_active":"2024-03-23T15:47:27.454952+0000","last_became_peered":"2024-03-23T15:47:27.454952+0000","last_unstale":"2024-03-23T16:01:53.072851+0000","last_undegraded":"2024-03-23T16:01:53.072851+0000","last_fullsized":"2024-03-23T16:01:53.072851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:53.072780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:53.072780+0000","last_clean_scrub_stamp":"2024-03-23T16:01:53.072780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:17:18.405981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"24'16","reported_seq":51,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.065500+0000","last_change":"2024-03-23T16:01:36.065535+0000","last_active":"2024-03-23T16:01:36.065500+0000","last_peered":"2024-03-23T16:01:36.065500+0000","last_clean":"2024-03-23T16:01:36.065500+0000","last_became_active":"2024-03-23T15:47:26.146903+0000","last_became_peered":"2024-03-23T15:47:26.146903+0000","last_unstale":"2024-03-23T16:01:36.065500+0000","last_undegraded":"2024-03-23T16:01:36.065500+0000","last_fullsized":"2024-03-23T16:01:36.065500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'16","last_scrub_stamp":"2024-03-23T16:01:36.065425+0000","last_deep_scrub":"24'16","last_deep_scrub_stamp":"2024-03-23T16:01:36.065425+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.065425+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:06:57.525502+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.371360+0000","last_change":"2024-03-23T16:01:40.371360+0000","last_active":"2024-03-23T16:01:40.371360+0000","last_peered":"2024-03-23T16:01:40.371360+0000","last_clean":"2024-03-23T16:01:40.371360+0000","last_became_active":"2024-03-23T15:47:27.356494+0000","last_became_peered":"2024-03-23T15:47:27.356494+0000","last_unstale":"2024-03-23T16:01:40.371360+0000","last_undegraded":"2024-03-23T16:01:40.371360+0000","last_fullsized":"2024-03-23T16:01:40.371360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:40.371224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:40.371224+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.371224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:04.962948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"24'15","reported_seq":50,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.416915+0000","last_change":"2024-03-23T16:01:41.416972+0000","last_active":"2024-03-23T16:01:41.416915+0000","last_peered":"2024-03-23T16:01:41.416915+0000","last_clean":"2024-03-23T16:01:41.416915+0000","last_became_active":"2024-03-23T15:47:26.151975+0000","last_became_peered":"2024-03-23T15:47:26.151975+0000","last_unstale":"2024-03-23T16:01:41.416915+0000","last_undegraded":"2024-03-23T16:01:41.416915+0000","last_fullsized":"2024-03-23T16:01:41.416915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'15","last_scrub_stamp":"2024-03-23T16:01:41.416817+0000","last_deep_scrub":"24'15","last_deep_scrub_stamp":"2024-03-23T16:01:41.416817+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.416817+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:30:12.085467+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:47.881167+0000","last_change":"2024-03-23T16:01:47.881188+0000","last_active":"2024-03-23T16:01:47.881167+0000","last_peered":"2024-03-23T16:01:47.881167+0000","last_clean":"2024-03-23T16:01:47.881167+0000","last_became_active":"2024-03-23T15:47:27.456057+0000","last_became_peered":"2024-03-23T15:47:27.456057+0000","last_unstale":"2024-03-23T16:01:47.881167+0000","last_undegraded":"2024-03-23T16:01:47.881167+0000","last_fullsized":"2024-03-23T16:01:47.881167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:47.881100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:47.881100+0000","last_clean_scrub_stamp":"2024-03-23T16:01:47.881100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:20:58.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"24'9","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.107311+0000","last_change":"2024-03-23T16:01:41.107357+0000","last_active":"2024-03-23T16:01:41.107311+0000","last_peered":"2024-03-23T16:01:41.107311+0000","last_clean":"2024-03-23T16:01:41.107311+0000","last_became_active":"2024-03-23T15:47:26.146330+0000","last_became_peered":"2024-03-23T15:47:26.146330+0000","last_unstale":"2024-03-23T16:01:41.107311+0000","last_undegraded":"2024-03-23T16:01:41.107311+0000","last_fullsized":"2024-03-23T16:01:41.107311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-23T16:01:41.107242+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-23T16:01:41.107242+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.107242+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:24.351834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:37.141195+0000","last_change":"2024-03-23T16:01:37.141217+0000","last_active":"2024-03-23T16:01:37.141195+0000","last_peered":"2024-03-23T16:01:37.141195+0000","last_clean":"2024-03-23T16:01:37.141195+0000","last_became_active":"2024-03-23T15:47:27.451462+0000","last_became_peered":"2024-03-23T15:47:27.451462+0000","last_unstale":"2024-03-23T16:01:37.141195+0000","last_undegraded":"2024-03-23T16:01:37.141195+0000","last_fullsized":"2024-03-23T16:01:37.141195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:37.141126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:37.141126+0000","last_clean_scrub_stamp":"2024-03-23T16:01:37.141126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:00.163367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"24'8","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.322779+0000","last_change":"2024-03-23T16:01:38.322823+0000","last_active":"2024-03-23T16:01:38.322779+0000","last_peered":"2024-03-23T16:01:38.322779+0000","last_clean":"2024-03-23T16:01:38.322779+0000","last_became_active":"2024-03-23T15:47:26.152854+0000","last_became_peered":"2024-03-23T15:47:26.152854+0000","last_unstale":"2024-03-23T16:01:38.322779+0000","last_undegraded":"2024-03-23T16:01:38.322779+0000","last_fullsized":"2024-03-23T16:01:38.322779+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-23T16:01:38.322685+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-23T16:01:38.322685+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.322685+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:51:04.453860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:37.740552+0000","last_change":"2024-03-23T16:01:37.740571+0000","last_active":"2024-03-23T16:01:37.740552+0000","last_peered":"2024-03-23T16:01:37.740552+0000","last_clean":"2024-03-23T16:01:37.740552+0000","last_became_active":"2024-03-23T15:47:27.453854+0000","last_became_peered":"2024-03-23T15:47:27.453854+0000","last_unstale":"2024-03-23T16:01:37.740552+0000","last_undegraded":"2024-03-23T16:01:37.740552+0000","last_fullsized":"2024-03-23T16:01:37.740552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:37.740467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:37.740467+0000","last_clean_scrub_stamp":"2024-03-23T16:01:37.740467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:23:26.839102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"24'25","reported_seq":60,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:36.718337+0000","last_change":"2024-03-23T16:01:36.718369+0000","last_active":"2024-03-23T16:01:36.718337+0000","last_peered":"2024-03-23T16:01:36.718337+0000","last_clean":"2024-03-23T16:01:36.718337+0000","last_became_active":"2024-03-23T15:47:26.150033+0000","last_became_peered":"2024-03-23T15:47:26.150033+0000","last_unstale":"2024-03-23T16:01:36.718337+0000","last_undegraded":"2024-03-23T16:01:36.718337+0000","last_fullsized":"2024-03-23T16:01:36.718337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'25","last_scrub_stamp":"2024-03-23T16:01:36.718277+0000","last_deep_scrub":"24'25","last_deep_scrub_stamp":"2024-03-23T16:01:36.718277+0000","last_clean_scrub_stamp":"2024-03-23T16:01:36.718277+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:24:02.977088+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:52.075050+0000","last_change":"2024-03-23T16:01:52.075073+0000","last_active":"2024-03-23T16:01:52.075050+0000","last_peered":"2024-03-23T16:01:52.075050+0000","last_clean":"2024-03-23T16:01:52.075050+0000","last_became_active":"2024-03-23T15:47:27.454488+0000","last_became_peered":"2024-03-23T15:47:27.454488+0000","last_unstale":"2024-03-23T16:01:52.075050+0000","last_undegraded":"2024-03-23T16:01:52.075050+0000","last_fullsized":"2024-03-23T16:01:52.075050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:52.074963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:52.074963+0000","last_clean_scrub_stamp":"2024-03-23T16:01:52.074963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:44:22.460814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"24'4","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.366349+0000","last_change":"2024-03-23T16:01:39.366394+0000","last_active":"2024-03-23T16:01:39.366349+0000","last_peered":"2024-03-23T16:01:39.366349+0000","last_clean":"2024-03-23T16:01:39.366349+0000","last_became_active":"2024-03-23T15:47:26.154831+0000","last_became_peered":"2024-03-23T15:47:26.154831+0000","last_unstale":"2024-03-23T16:01:39.366349+0000","last_undegraded":"2024-03-23T16:01:39.366349+0000","last_fullsized":"2024-03-23T16:01:39.366349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4","last_scrub_stamp":"2024-03-23T16:01:39.366256+0000","last_deep_scrub":"24'4","last_deep_scrub_stamp":"2024-03-23T16:01:39.366256+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.366256+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:56:26.957508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.041319+0000","last_change":"2024-03-23T16:01:40.041354+0000","last_active":"2024-03-23T16:01:40.041319+0000","last_peered":"2024-03-23T16:01:40.041319+0000","last_clean":"2024-03-23T16:01:40.041319+0000","last_became_active":"2024-03-23T15:47:27.448152+0000","last_became_peered":"2024-03-23T15:47:27.448152+0000","last_unstale":"2024-03-23T16:01:40.041319+0000","last_undegraded":"2024-03-23T16:01:40.041319+0000","last_fullsized":"2024-03-23T16:01:40.041319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:40.041241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:40.041241+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.041241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:40:53.462990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"24'7","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:37.683240+0000","last_change":"2024-03-23T16:01:37.683287+0000","last_active":"2024-03-23T16:01:37.683240+0000","last_peered":"2024-03-23T16:01:37.683240+0000","last_clean":"2024-03-23T16:01:37.683240+0000","last_became_active":"2024-03-23T15:47:26.151699+0000","last_became_peered":"2024-03-23T15:47:26.151699+0000","last_unstale":"2024-03-23T16:01:37.683240+0000","last_undegraded":"2024-03-23T16:01:37.683240+0000","last_fullsized":"2024-03-23T16:01:37.683240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-23T16:01:37.683140+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-23T16:01:37.683140+0000","last_clean_scrub_stamp":"2024-03-23T16:01:37.683140+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:09:55.687480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:45.522455+0000","last_change":"2024-03-23T16:01:45.522474+0000","last_active":"2024-03-23T16:01:45.522455+0000","last_peered":"2024-03-23T16:01:45.522455+0000","last_clean":"2024-03-23T16:01:45.522455+0000","last_became_active":"2024-03-23T15:47:27.451161+0000","last_became_peered":"2024-03-23T15:47:27.451161+0000","last_unstale":"2024-03-23T16:01:45.522455+0000","last_undegraded":"2024-03-23T16:01:45.522455+0000","last_fullsized":"2024-03-23T16:01:45.522455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:45.522385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:45.522385+0000","last_clean_scrub_stamp":"2024-03-23T16:01:45.522385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:59:30.290315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'4","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.303219+0000","last_change":"2024-03-23T16:01:39.303264+0000","last_active":"2024-03-23T16:01:39.303219+0000","last_peered":"2024-03-23T16:01:39.303219+0000","last_clean":"2024-03-23T16:01:39.303219+0000","last_became_active":"2024-03-23T15:47:26.147050+0000","last_became_peered":"2024-03-23T15:47:26.147050+0000","last_unstale":"2024-03-23T16:01:39.303219+0000","last_undegraded":"2024-03-23T16:01:39.303219+0000","last_fullsized":"2024-03-23T16:01:39.303219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4","last_scrub_stamp":"2024-03-23T16:01:39.303115+0000","last_deep_scrub":"24'4","last_deep_scrub_stamp":"2024-03-23T16:01:39.303115+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.303115+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:02.137050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:44.511001+0000","last_change":"2024-03-23T16:01:44.511020+0000","last_active":"2024-03-23T16:01:44.511001+0000","last_peered":"2024-03-23T16:01:44.511001+0000","last_clean":"2024-03-23T16:01:44.511001+0000","last_became_active":"2024-03-23T15:47:27.448628+0000","last_became_peered":"2024-03-23T15:47:27.448628+0000","last_unstale":"2024-03-23T16:01:44.511001+0000","last_undegraded":"2024-03-23T16:01:44.511001+0000","last_fullsized":"2024-03-23T16:01:44.511001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:44.510937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:44.510937+0000","last_clean_scrub_stamp":"2024-03-23T16:01:44.510937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:47:53.306776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"24'8","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.911363+0000","last_change":"2024-03-23T16:01:40.911407+0000","last_active":"2024-03-23T16:01:40.911363+0000","last_peered":"2024-03-23T16:01:40.911363+0000","last_clean":"2024-03-23T16:01:40.911363+0000","last_became_active":"2024-03-23T15:47:26.152459+0000","last_became_peered":"2024-03-23T15:47:26.152459+0000","last_unstale":"2024-03-23T16:01:40.911363+0000","last_undegraded":"2024-03-23T16:01:40.911363+0000","last_fullsized":"2024-03-23T16:01:40.911363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-23T16:01:40.911290+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-23T16:01:40.911290+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.911290+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:24:07.170102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:56.111669+0000","last_change":"2024-03-23T16:01:56.111708+0000","last_active":"2024-03-23T16:01:56.111669+0000","last_peered":"2024-03-23T16:01:56.111669+0000","last_clean":"2024-03-23T16:01:56.111669+0000","last_became_active":"2024-03-23T15:47:27.457072+0000","last_became_peered":"2024-03-23T15:47:27.457072+0000","last_unstale":"2024-03-23T16:01:56.111669+0000","last_undegraded":"2024-03-23T16:01:56.111669+0000","last_fullsized":"2024-03-23T16:01:56.111669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:56.111589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:56.111589+0000","last_clean_scrub_stamp":"2024-03-23T16:01:56.111589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:37:14.192727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"24'119","reported_seq":195,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:34.761995+0000","last_change":"2024-03-23T16:01:34.762020+0000","last_active":"2024-03-23T16:01:34.761995+0000","last_peered":"2024-03-23T16:01:34.761995+0000","last_clean":"2024-03-23T16:01:34.761995+0000","last_became_active":"2024-03-23T15:47:14.622693+0000","last_became_peered":"2024-03-23T15:47:14.622693+0000","last_unstale":"2024-03-23T16:01:34.761995+0000","last_undegraded":"2024-03-23T16:01:34.761995+0000","last_fullsized":"2024-03-23T16:01:34.761995+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'119","last_scrub_stamp":"2024-03-23T16:01:34.761938+0000","last_deep_scrub":"24'119","last_deep_scrub_stamp":"2024-03-23T16:01:34.761938+0000","last_clean_scrub_stamp":"2024-03-23T16:01:34.761938+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:35.812801+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"24'11","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:45.065408+0000","last_change":"2024-03-23T16:01:45.065438+0000","last_active":"2024-03-23T16:01:45.065408+0000","last_peered":"2024-03-23T16:01:45.065408+0000","last_clean":"2024-03-23T16:01:45.065408+0000","last_became_active":"2024-03-23T15:47:26.148873+0000","last_became_peered":"2024-03-23T15:47:26.148873+0000","last_unstale":"2024-03-23T16:01:45.065408+0000","last_undegraded":"2024-03-23T16:01:45.065408+0000","last_fullsized":"2024-03-23T16:01:45.065408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-23T16:01:45.065347+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-23T16:01:45.065347+0000","last_clean_scrub_stamp":"2024-03-23T16:01:45.065347+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:15:29.386923+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:47.675583+0000","last_change":"2024-03-23T16:01:47.675619+0000","last_active":"2024-03-23T16:01:47.675583+0000","last_peered":"2024-03-23T16:01:47.675583+0000","last_clean":"2024-03-23T16:01:47.675583+0000","last_became_active":"2024-03-23T15:47:27.450920+0000","last_became_peered":"2024-03-23T15:47:27.450920+0000","last_unstale":"2024-03-23T16:01:47.675583+0000","last_undegraded":"2024-03-23T16:01:47.675583+0000","last_fullsized":"2024-03-23T16:01:47.675583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:47.675462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:47.675462+0000","last_clean_scrub_stamp":"2024-03-23T16:01:47.675462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:58:59.579475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"24'6","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.109975+0000","last_change":"2024-03-23T16:01:42.110019+0000","last_active":"2024-03-23T16:01:42.109975+0000","last_peered":"2024-03-23T16:01:42.109975+0000","last_clean":"2024-03-23T16:01:42.109975+0000","last_became_active":"2024-03-23T15:47:26.147265+0000","last_became_peered":"2024-03-23T15:47:26.147265+0000","last_unstale":"2024-03-23T16:01:42.109975+0000","last_undegraded":"2024-03-23T16:01:42.109975+0000","last_fullsized":"2024-03-23T16:01:42.109975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-23T16:01:42.109885+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-23T16:01:42.109885+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.109885+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:47:50.905724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:46.897886+0000","last_change":"2024-03-23T16:01:46.897923+0000","last_active":"2024-03-23T16:01:46.897886+0000","last_peered":"2024-03-23T16:01:46.897886+0000","last_clean":"2024-03-23T16:01:46.897886+0000","last_became_active":"2024-03-23T15:47:27.449195+0000","last_became_peered":"2024-03-23T15:47:27.449195+0000","last_unstale":"2024-03-23T16:01:46.897886+0000","last_undegraded":"2024-03-23T16:01:46.897886+0000","last_fullsized":"2024-03-23T16:01:46.897886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:46.897779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:46.897779+0000","last_clean_scrub_stamp":"2024-03-23T16:01:46.897779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:06:13.832521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"24'9","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.474385+0000","last_change":"2024-03-23T16:01:39.474411+0000","last_active":"2024-03-23T16:01:39.474385+0000","last_peered":"2024-03-23T16:01:39.474385+0000","last_clean":"2024-03-23T16:01:39.474385+0000","last_became_active":"2024-03-23T15:47:26.146620+0000","last_became_peered":"2024-03-23T15:47:26.146620+0000","last_unstale":"2024-03-23T16:01:39.474385+0000","last_undegraded":"2024-03-23T16:01:39.474385+0000","last_fullsized":"2024-03-23T16:01:39.474385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-23T16:01:39.474324+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-23T16:01:39.474324+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.474324+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:19:51.850377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:52.263212+0000","last_change":"2024-03-23T16:01:52.263248+0000","last_active":"2024-03-23T16:01:52.263212+0000","last_peered":"2024-03-23T16:01:52.263212+0000","last_clean":"2024-03-23T16:01:52.263212+0000","last_became_active":"2024-03-23T15:47:27.450624+0000","last_became_peered":"2024-03-23T15:47:27.450624+0000","last_unstale":"2024-03-23T16:01:52.263212+0000","last_undegraded":"2024-03-23T16:01:52.263212+0000","last_fullsized":"2024-03-23T16:01:52.263212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:52.263132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:52.263132+0000","last_clean_scrub_stamp":"2024-03-23T16:01:52.263132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:59:51.742632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'7","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:47.057533+0000","last_change":"2024-03-23T16:01:47.057557+0000","last_active":"2024-03-23T16:01:47.057533+0000","last_peered":"2024-03-23T16:01:47.057533+0000","last_clean":"2024-03-23T16:01:47.057533+0000","last_became_active":"2024-03-23T15:47:26.149861+0000","last_became_peered":"2024-03-23T15:47:26.149861+0000","last_unstale":"2024-03-23T16:01:47.057533+0000","last_undegraded":"2024-03-23T16:01:47.057533+0000","last_fullsized":"2024-03-23T16:01:47.057533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-23T16:01:47.057475+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-23T16:01:47.057475+0000","last_clean_scrub_stamp":"2024-03-23T16:01:47.057475+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:49.256851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.485857+0000","last_change":"2024-03-23T16:01:43.485890+0000","last_active":"2024-03-23T16:01:43.485857+0000","last_peered":"2024-03-23T16:01:43.485857+0000","last_clean":"2024-03-23T16:01:43.485857+0000","last_became_active":"2024-03-23T15:47:27.448865+0000","last_became_peered":"2024-03-23T15:47:27.448865+0000","last_unstale":"2024-03-23T16:01:43.485857+0000","last_undegraded":"2024-03-23T16:01:43.485857+0000","last_fullsized":"2024-03-23T16:01:43.485857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:43.485756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:43.485756+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.485756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:21:26.221345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.293085+0000","last_change":"2024-03-23T16:01:40.293127+0000","last_active":"2024-03-23T16:01:40.293085+0000","last_peered":"2024-03-23T16:01:40.293085+0000","last_clean":"2024-03-23T16:01:40.293085+0000","last_became_active":"2024-03-23T15:47:26.149883+0000","last_became_peered":"2024-03-23T15:47:26.149883+0000","last_unstale":"2024-03-23T16:01:40.293085+0000","last_undegraded":"2024-03-23T16:01:40.293085+0000","last_fullsized":"2024-03-23T16:01:40.293085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-23T16:01:40.292969+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-23T16:01:40.292969+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.292969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:16:05.939949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.051713+0000","last_change":"2024-03-23T16:01:38.051733+0000","last_active":"2024-03-23T16:01:38.051713+0000","last_peered":"2024-03-23T16:01:38.051713+0000","last_clean":"2024-03-23T16:01:38.051713+0000","last_became_active":"2024-03-23T15:47:27.442876+0000","last_became_peered":"2024-03-23T15:47:27.442876+0000","last_unstale":"2024-03-23T16:01:38.051713+0000","last_undegraded":"2024-03-23T16:01:38.051713+0000","last_fullsized":"2024-03-23T16:01:38.051713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:38.051649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:38.051649+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.051649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:23:27.150284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"24'7","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:44.075920+0000","last_change":"2024-03-23T16:01:44.075946+0000","last_active":"2024-03-23T16:01:44.075920+0000","last_peered":"2024-03-23T16:01:44.075920+0000","last_clean":"2024-03-23T16:01:44.075920+0000","last_became_active":"2024-03-23T15:47:26.147547+0000","last_became_peered":"2024-03-23T15:47:26.147547+0000","last_unstale":"2024-03-23T16:01:44.075920+0000","last_undegraded":"2024-03-23T16:01:44.075920+0000","last_fullsized":"2024-03-23T16:01:44.075920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-23T16:01:44.075857+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-23T16:01:44.075857+0000","last_clean_scrub_stamp":"2024-03-23T16:01:44.075857+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:18:30.691782+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.858284+0000","last_change":"2024-03-23T16:01:43.858318+0000","last_active":"2024-03-23T16:01:43.858284+0000","last_peered":"2024-03-23T16:01:43.858284+0000","last_clean":"2024-03-23T16:01:43.858284+0000","last_became_active":"2024-03-23T15:47:27.382857+0000","last_became_peered":"2024-03-23T15:47:27.382857+0000","last_unstale":"2024-03-23T16:01:43.858284+0000","last_undegraded":"2024-03-23T16:01:43.858284+0000","last_fullsized":"2024-03-23T16:01:43.858284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:43.858173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:43.858173+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.858173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:45.242539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"24'11","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.668702+0000","last_change":"2024-03-23T16:01:38.668753+0000","last_active":"2024-03-23T16:01:38.668702+0000","last_peered":"2024-03-23T16:01:38.668702+0000","last_clean":"2024-03-23T16:01:38.668702+0000","last_became_active":"2024-03-23T15:47:26.153666+0000","last_became_peered":"2024-03-23T15:47:26.153666+0000","last_unstale":"2024-03-23T16:01:38.668702+0000","last_undegraded":"2024-03-23T16:01:38.668702+0000","last_fullsized":"2024-03-23T16:01:38.668702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-23T16:01:38.668600+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-23T16:01:38.668600+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.668600+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:23:58.484346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:41.643177+0000","last_change":"2024-03-23T16:01:41.643214+0000","last_active":"2024-03-23T16:01:41.643177+0000","last_peered":"2024-03-23T16:01:41.643177+0000","last_clean":"2024-03-23T16:01:41.643177+0000","last_became_active":"2024-03-23T15:47:27.438399+0000","last_became_peered":"2024-03-23T15:47:27.438399+0000","last_unstale":"2024-03-23T16:01:41.643177+0000","last_undegraded":"2024-03-23T16:01:41.643177+0000","last_fullsized":"2024-03-23T16:01:41.643177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:41.643060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:41.643060+0000","last_clean_scrub_stamp":"2024-03-23T16:01:41.643060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:04.476994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"24'11","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.103965+0000","last_change":"2024-03-23T16:01:40.104015+0000","last_active":"2024-03-23T16:01:40.103965+0000","last_peered":"2024-03-23T16:01:40.103965+0000","last_clean":"2024-03-23T16:01:40.103965+0000","last_became_active":"2024-03-23T15:47:26.150997+0000","last_became_peered":"2024-03-23T15:47:26.150997+0000","last_unstale":"2024-03-23T16:01:40.103965+0000","last_undegraded":"2024-03-23T16:01:40.103965+0000","last_fullsized":"2024-03-23T16:01:40.103965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-23T16:01:40.103859+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-23T16:01:40.103859+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.103859+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:58:52.007872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:44.677870+0000","last_change":"2024-03-23T16:01:44.677908+0000","last_active":"2024-03-23T16:01:44.677870+0000","last_peered":"2024-03-23T16:01:44.677870+0000","last_clean":"2024-03-23T16:01:44.677870+0000","last_became_active":"2024-03-23T15:47:27.448241+0000","last_became_peered":"2024-03-23T15:47:27.448241+0000","last_unstale":"2024-03-23T16:01:44.677870+0000","last_undegraded":"2024-03-23T16:01:44.677870+0000","last_fullsized":"2024-03-23T16:01:44.677870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:44.677755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:44.677755+0000","last_clean_scrub_stamp":"2024-03-23T16:01:44.677755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:56:32.269007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'3","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:37.335424+0000","last_change":"2024-03-23T16:01:37.335465+0000","last_active":"2024-03-23T16:01:37.335424+0000","last_peered":"2024-03-23T16:01:37.335424+0000","last_clean":"2024-03-23T16:01:37.335424+0000","last_became_active":"2024-03-23T15:47:26.153053+0000","last_became_peered":"2024-03-23T15:47:26.153053+0000","last_unstale":"2024-03-23T16:01:37.335424+0000","last_undegraded":"2024-03-23T16:01:37.335424+0000","last_fullsized":"2024-03-23T16:01:37.335424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3","last_scrub_stamp":"2024-03-23T16:01:37.335345+0000","last_deep_scrub":"24'3","last_deep_scrub_stamp":"2024-03-23T16:01:37.335345+0000","last_clean_scrub_stamp":"2024-03-23T16:01:37.335345+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:06:04.270088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.101643+0000","last_change":"2024-03-23T16:01:38.101662+0000","last_active":"2024-03-23T16:01:38.101643+0000","last_peered":"2024-03-23T16:01:38.101643+0000","last_clean":"2024-03-23T16:01:38.101643+0000","last_became_active":"2024-03-23T15:47:27.450336+0000","last_became_peered":"2024-03-23T15:47:27.450336+0000","last_unstale":"2024-03-23T16:01:38.101643+0000","last_undegraded":"2024-03-23T16:01:38.101643+0000","last_fullsized":"2024-03-23T16:01:38.101643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:38.101583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:38.101583+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.101583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:02.693306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"24'11","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.679368+0000","last_change":"2024-03-23T16:01:39.679413+0000","last_active":"2024-03-23T16:01:39.679368+0000","last_peered":"2024-03-23T16:01:39.679368+0000","last_clean":"2024-03-23T16:01:39.679368+0000","last_became_active":"2024-03-23T15:47:26.153435+0000","last_became_peered":"2024-03-23T15:47:26.153435+0000","last_unstale":"2024-03-23T16:01:39.679368+0000","last_undegraded":"2024-03-23T16:01:39.679368+0000","last_fullsized":"2024-03-23T16:01:39.679368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-23T16:01:39.679271+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-23T16:01:39.679271+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.679271+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:19.518164+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:43.693227+0000","last_change":"2024-03-23T16:01:43.693261+0000","last_active":"2024-03-23T16:01:43.693227+0000","last_peered":"2024-03-23T16:01:43.693227+0000","last_clean":"2024-03-23T16:01:43.693227+0000","last_became_active":"2024-03-23T15:47:27.448210+0000","last_became_peered":"2024-03-23T15:47:27.448210+0000","last_unstale":"2024-03-23T16:01:43.693227+0000","last_undegraded":"2024-03-23T16:01:43.693227+0000","last_fullsized":"2024-03-23T16:01:43.693227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:43.693154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:43.693154+0000","last_clean_scrub_stamp":"2024-03-23T16:01:43.693154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:47:10.390977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"24'3","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.919060+0000","last_change":"2024-03-23T16:01:38.919100+0000","last_active":"2024-03-23T16:01:38.919060+0000","last_peered":"2024-03-23T16:01:38.919060+0000","last_clean":"2024-03-23T16:01:38.919060+0000","last_became_active":"2024-03-23T15:47:26.150894+0000","last_became_peered":"2024-03-23T15:47:26.150894+0000","last_unstale":"2024-03-23T16:01:38.919060+0000","last_undegraded":"2024-03-23T16:01:38.919060+0000","last_fullsized":"2024-03-23T16:01:38.919060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3","last_scrub_stamp":"2024-03-23T16:01:38.918977+0000","last_deep_scrub":"24'3","last_deep_scrub_stamp":"2024-03-23T16:01:38.918977+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.918977+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:10:47.567819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:42.461939+0000","last_change":"2024-03-23T16:01:42.461972+0000","last_active":"2024-03-23T16:01:42.461939+0000","last_peered":"2024-03-23T16:01:42.461939+0000","last_clean":"2024-03-23T16:01:42.461939+0000","last_became_active":"2024-03-23T15:47:27.447981+0000","last_became_peered":"2024-03-23T15:47:27.447981+0000","last_unstale":"2024-03-23T16:01:42.461939+0000","last_undegraded":"2024-03-23T16:01:42.461939+0000","last_fullsized":"2024-03-23T16:01:42.461939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:42.461830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:42.461830+0000","last_clean_scrub_stamp":"2024-03-23T16:01:42.461830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:10:51.110672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"24'6","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:46.029458+0000","last_change":"2024-03-23T16:01:46.029484+0000","last_active":"2024-03-23T16:01:46.029458+0000","last_peered":"2024-03-23T16:01:46.029458+0000","last_clean":"2024-03-23T16:01:46.029458+0000","last_became_active":"2024-03-23T15:47:26.148393+0000","last_became_peered":"2024-03-23T15:47:26.148393+0000","last_unstale":"2024-03-23T16:01:46.029458+0000","last_undegraded":"2024-03-23T16:01:46.029458+0000","last_fullsized":"2024-03-23T16:01:46.029458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-23T16:01:46.029398+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-23T16:01:46.029398+0000","last_clean_scrub_stamp":"2024-03-23T16:01:46.029398+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:27:46.068848+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:55.067014+0000","last_change":"2024-03-23T16:01:55.067032+0000","last_active":"2024-03-23T16:01:55.067014+0000","last_peered":"2024-03-23T16:01:55.067014+0000","last_clean":"2024-03-23T16:01:55.067014+0000","last_became_active":"2024-03-23T15:47:27.456475+0000","last_became_peered":"2024-03-23T15:47:27.456475+0000","last_unstale":"2024-03-23T16:01:55.067014+0000","last_undegraded":"2024-03-23T16:01:55.067014+0000","last_fullsized":"2024-03-23T16:01:55.067014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:55.066952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:55.066952+0000","last_clean_scrub_stamp":"2024-03-23T16:01:55.066952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:28:49.826159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:50.099478+0000","last_change":"2024-03-23T16:01:50.099497+0000","last_active":"2024-03-23T16:01:50.099478+0000","last_peered":"2024-03-23T16:01:50.099478+0000","last_clean":"2024-03-23T16:01:50.099478+0000","last_became_active":"2024-03-23T15:47:27.452961+0000","last_became_peered":"2024-03-23T15:47:27.452961+0000","last_unstale":"2024-03-23T16:01:50.099478+0000","last_undegraded":"2024-03-23T16:01:50.099478+0000","last_fullsized":"2024-03-23T16:01:50.099478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:50.099416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:50.099416+0000","last_clean_scrub_stamp":"2024-03-23T16:01:50.099416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:02:50.671534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"24'8","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:35.301725+0000","last_change":"2024-03-23T16:01:35.301779+0000","last_active":"2024-03-23T16:01:35.301725+0000","last_peered":"2024-03-23T16:01:35.301725+0000","last_clean":"2024-03-23T16:01:35.301725+0000","last_became_active":"2024-03-23T15:47:26.152564+0000","last_became_peered":"2024-03-23T15:47:26.152564+0000","last_unstale":"2024-03-23T16:01:35.301725+0000","last_undegraded":"2024-03-23T16:01:35.301725+0000","last_fullsized":"2024-03-23T16:01:35.301725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-23T16:01:35.301625+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-23T16:01:35.301625+0000","last_clean_scrub_stamp":"2024-03-23T16:01:35.301625+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:25:55.415003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:39.687858+0000","last_change":"2024-03-23T16:01:39.687892+0000","last_active":"2024-03-23T16:01:39.687858+0000","last_peered":"2024-03-23T16:01:39.687858+0000","last_clean":"2024-03-23T16:01:39.687858+0000","last_became_active":"2024-03-23T15:47:27.455102+0000","last_became_peered":"2024-03-23T15:47:27.455102+0000","last_unstale":"2024-03-23T16:01:39.687858+0000","last_undegraded":"2024-03-23T16:01:39.687858+0000","last_fullsized":"2024-03-23T16:01:39.687858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:39.687758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:39.687758+0000","last_clean_scrub_stamp":"2024-03-23T16:01:39.687758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:51:40.253108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"24'19","reported_seq":53,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:34.278181+0000","last_change":"2024-03-23T16:01:34.278216+0000","last_active":"2024-03-23T16:01:34.278181+0000","last_peered":"2024-03-23T16:01:34.278181+0000","last_clean":"2024-03-23T16:01:34.278181+0000","last_became_active":"2024-03-23T15:47:26.151853+0000","last_became_peered":"2024-03-23T15:47:26.151853+0000","last_unstale":"2024-03-23T16:01:34.278181+0000","last_undegraded":"2024-03-23T16:01:34.278181+0000","last_fullsized":"2024-03-23T16:01:34.278181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'19","last_scrub_stamp":"2024-03-23T16:01:34.278119+0000","last_deep_scrub":"24'19","last_deep_scrub_stamp":"2024-03-23T16:01:34.278119+0000","last_clean_scrub_stamp":"2024-03-23T16:01:34.278119+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:22:35.662485+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:50.299727+0000","last_change":"2024-03-23T16:01:50.299746+0000","last_active":"2024-03-23T16:01:50.299727+0000","last_peered":"2024-03-23T16:01:50.299727+0000","last_clean":"2024-03-23T16:01:50.299727+0000","last_became_active":"2024-03-23T15:47:27.450296+0000","last_became_peered":"2024-03-23T15:47:27.450296+0000","last_unstale":"2024-03-23T16:01:50.299727+0000","last_undegraded":"2024-03-23T16:01:50.299727+0000","last_fullsized":"2024-03-23T16:01:50.299727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:50.299664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:50.299664+0000","last_clean_scrub_stamp":"2024-03-23T16:01:50.299664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:03:30.225736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:38.270473+0000","last_change":"2024-03-23T16:01:38.270524+0000","last_active":"2024-03-23T16:01:38.270473+0000","last_peered":"2024-03-23T16:01:38.270473+0000","last_clean":"2024-03-23T16:01:38.270473+0000","last_became_active":"2024-03-23T15:47:26.144391+0000","last_became_peered":"2024-03-23T15:47:26.144391+0000","last_unstale":"2024-03-23T16:01:38.270473+0000","last_undegraded":"2024-03-23T16:01:38.270473+0000","last_fullsized":"2024-03-23T16:01:38.270473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'2","last_scrub_stamp":"2024-03-23T16:01:38.270355+0000","last_deep_scrub":"21'2","last_deep_scrub_stamp":"2024-03-23T16:01:38.270355+0000","last_clean_scrub_stamp":"2024-03-23T16:01:38.270355+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:31.019892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:48.254177+0000","last_change":"2024-03-23T16:01:48.254214+0000","last_active":"2024-03-23T16:01:48.254177+0000","last_peered":"2024-03-23T16:01:48.254177+0000","last_clean":"2024-03-23T16:01:48.254177+0000","last_became_active":"2024-03-23T15:47:27.450091+0000","last_became_peered":"2024-03-23T15:47:27.450091+0000","last_unstale":"2024-03-23T16:01:48.254177+0000","last_undegraded":"2024-03-23T16:01:48.254177+0000","last_fullsized":"2024-03-23T16:01:48.254177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T16:01:48.254076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T16:01:48.254076+0000","last_clean_scrub_stamp":"2024-03-23T16:01:48.254076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:56:15.054256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"24'19","reported_seq":54,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-23T16:01:40.635293+0000","last_change":"2024-03-23T16:01:40.635347+0000","last_active":"2024-03-23T16:01:40.635293+0000","last_peered":"2024-03-23T16:01:40.635293+0000","last_clean":"2024-03-23T16:01:40.635293+0000","last_became_active":"2024-03-23T15:47:26.154777+0000","last_became_peered":"2024-03-23T15:47:26.154777+0000","last_unstale":"2024-03-23T16:01:40.635293+0000","last_undegraded":"2024-03-23T16:01:40.635293+0000","last_fullsized":"2024-03-23T16:01:40.635293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'19","last_scrub_stamp":"2024-03-23T16:01:40.635197+0000","last_deep_scrub":"24'19","last_deep_scrub_stamp":"2024-03-23T16:01:40.635197+0000","last_clean_scrub_stamp":"2024-03-23T16:01:40.635197+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:26:00.748574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65216,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575029,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":290816,"data_stored":95068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575028,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":96394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575028,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":344064,"data_stored":685208,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":97464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575029,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":96736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":95694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":685796,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575029,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":96556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32608,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T16:02:00.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-23T16:02:00.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.608+0000 7f0eaedf4640 1 Processor -- start 2024-03-23T16:02:00.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.608+0000 7f0eaedf4640 1 -- start start 2024-03-23T16:02:00.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.609+0000 7f0eaedf4640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ea815acd0 0x7f0ea80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:02:00.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.609+0000 7f0eaedf4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea80cab80 con 0x7f0ea80ccc30 2024-03-23T16:02:00.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.609+0000 7f0eaedf4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea806da20 con 0x7f0ea80ced70 2024-03-23T16:02:00.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.609+0000 7f0eaedf4640 1 -- --> v2:172.21.15.129:3300/0 -- mon_getmap magic: 0 -- 0x7f0ea80cf690 con 0x7f0ea815acd0 2024-03-23T16:02:00.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.612+0000 7f0ead391640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ea815acd0 0x7f0ea80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:02:00.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.612+0000 7f0ead391640 1 --2- >> v2:172.21.15.129:3300/0 conn(0x7f0ea815acd0 0x7f0ea80cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58976/0 (socket says 172.21.15.129:58976) 2024-03-23T16:02:00.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.612+0000 7f0ead391640 1 -- 172.21.15.129:0/1854939097 learned_addr learned my addr 172.21.15.129:0/1854939097 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:02:00.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.616+0000 7f0ea77fe640 1 -- 172.21.15.129:0/1854939097 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851905898 0 0) 0x7f0ea806da20 con 0x7f0ea80ced70 2024-03-23T16:02:00.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.616+0000 7f0ea77fe640 1 -- 172.21.15.129:0/1854939097 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e98003520 con 0x7f0ea80ced70 2024-03-23T16:02:00.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.618+0000 7f0ead391640 1 -- 172.21.15.129:0/1854939097 >> v1:172.21.15.157:6789/0 conn(0x7f0ea80ced70 legacy=0x7f0ea80ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.618+0000 7f0ead391640 1 -- 172.21.15.129:0/1854939097 >> v1:172.21.15.129:6789/0 conn(0x7f0ea80ccc30 legacy=0x7f0ea815a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.618+0000 7f0ead391640 1 -- 172.21.15.129:0/1854939097 --> v2:172.21.15.129:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea806d460 con 0x7f0ea815acd0 2024-03-23T16:02:00.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.621+0000 7f0ead391640 1 --2- 172.21.15.129:0/1854939097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea815acd0 0x7f0ea80cf150 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009e50 tx=0x7f0ea002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a63b7b5c802cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:02:00.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.628+0000 7f0ea77fe640 1 -- 172.21.15.129:0/1854939097 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea002cbb0 con 0x7f0ea815acd0 2024-03-23T16:02:00.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.628+0000 7f0ea77fe640 1 -- 172.21.15.129:0/1854939097 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ea002cd10 con 0x7f0ea815acd0 2024-03-23T16:02:00.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.628+0000 7f0ea77fe640 1 -- 172.21.15.129:0/1854939097 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0034610 con 0x7f0ea815acd0 2024-03-23T16:02:00.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1854939097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea815acd0 msgr2=0x7f0ea80cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1854939097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea815acd0 0x7f0ea80cf150 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009e50 tx=0x7f0ea002c010 comp rx=0 tx=0).stop 2024-03-23T16:02:00.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1854939097 shutdown_connections 2024-03-23T16:02:00.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1854939097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea815acd0 0x7f0ea80cf150 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1854939097 >> 172.21.15.129:0/1854939097 conn(0x7f0ea80cfaa0 msgr2=0x7f0ea80cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:02:00.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1854939097 shutdown_connections 2024-03-23T16:02:00.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.629+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1854939097 wait complete. 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.630+0000 7f0eaedf4640 1 Processor -- start 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.630+0000 7f0eaedf4640 1 -- start start 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.631+0000 7f0eaedf4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea80ccc30 0x7f0ea8154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.631+0000 7f0eaedf4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 0x7f0ea80b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.631+0000 7f0eaedf4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ea815acd0 0x7f0ea8159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.631+0000 7f0eaedf4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea80b1880 con 0x7f0ea80ccc30 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.631+0000 7f0eaedf4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea814bee0 con 0x7f0ea815acd0 2024-03-23T16:02:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.631+0000 7f0eaedf4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea80b1680 con 0x7f0ea80ced70 2024-03-23T16:02:00.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.632+0000 7f0ead391640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ea815acd0 0x7f0ea8159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:02:00.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.632+0000 7f0ea7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 0x7f0ea80b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:02:00.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.632+0000 7f0ea7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 0x7f0ea80b1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.129:34102/0 (socket says 172.21.15.129:34102) 2024-03-23T16:02:00.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.632+0000 7f0ea7fff640 1 -- 172.21.15.129:0/1662223268 learned_addr learned my addr 172.21.15.129:0/1662223268 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-23T16:02:00.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.633+0000 7f0eacb90640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea80ccc30 0x7f0ea8154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:02:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.636+0000 7f0ea7fff640 1 -- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ea815acd0 msgr2=0x7f0ea8159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.636+0000 7f0ea7fff640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ea815acd0 0x7f0ea8159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.636+0000 7f0ea7fff640 1 -- 172.21.15.129:0/1662223268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea80ccc30 msgr2=0x7f0ea8154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.636+0000 7f0ea7fff640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea80ccc30 0x7f0ea8154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.636+0000 7f0ea7fff640 1 -- 172.21.15.129:0/1662223268 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea0009b30 con 0x7f0ea80ced70 2024-03-23T16:02:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.638+0000 7f0ea7fff640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 0x7f0ea80b1140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c430 tx=0x7f0e9400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:02:00.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.643+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e940052d0 con 0x7f0ea80ced70 2024-03-23T16:02:00.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.643+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e94005450 con 0x7f0ea80ced70 2024-03-23T16:02:00.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.643+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94007260 con 0x7f0ea80ced70 2024-03-23T16:02:00.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.643+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea8155460 con 0x7f0ea80ced70 2024-03-23T16:02:00.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.643+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea80af7b0 con 0x7f0ea80ced70 2024-03-23T16:02:00.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.644+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea814c020 con 0x7f0ea80ced70 2024-03-23T16:02:00.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.678+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0e940073c0 con 0x7f0ea80ced70 2024-03-23T16:02:00.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.678+0000 7f0ea57fa640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8c06b860 0x7f0e8c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T16:02:00.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.679+0000 7f0eacb90640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8c06b860 0x7f0e8c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T16:02:00.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.679+0000 7f0eacb90640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8c06b860 0x7f0e8c06dd20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0011e0 tx=0x7f0e9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T16:02:00.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.684+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5824+0+0 (secure 0 0 0) 0x7f0e94057f80 con 0x7f0ea80ced70 2024-03-23T16:02:00.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:00.951+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0e94054220 con 0x7f0ea80ced70 2024-03-23T16:02:01.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.133+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f0ea8155460 con 0x7f0ea80ced70 2024-03-23T16:02:01.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.378+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0e94054220 con 0x7f0ea80ced70 2024-03-23T16:02:01.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.378+0000 7f0ea57fa640 1 -- 172.21.15.129:0/1662223268 <== mon.2 v2:172.21.15.157:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f0ea8155460 con 0x7f0ea80ced70 2024-03-23T16:02:01.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8c06b860 msgr2=0x7f0e8c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:01.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8c06b860 0x7f0e8c06dd20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0011e0 tx=0x7f0e9c009040 comp rx=0 tx=0).stop 2024-03-23T16:02:01.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 msgr2=0x7f0ea80b1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T16:02:01.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 0x7f0ea80b1140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c430 tx=0x7f0e9400c8f0 comp rx=0 tx=0).stop 2024-03-23T16:02:01.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 shutdown_connections 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:6800/2961890009,v1:172.21.15.157:6801/2961890009] conn(0x7f0e8c06b860 0x7f0e8c06dd20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ea815acd0 0x7f0ea8159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0ea80ced70 0x7f0ea80b1140 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 --2- 172.21.15.129:0/1662223268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea80ccc30 0x7f0ea8154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 >> 172.21.15.129:0/1662223268 conn(0x7f0ea80cfaa0 msgr2=0x7f0ea814eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 shutdown_connections 2024-03-23T16:02:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-23T16:02:01.381+0000 7f0eaedf4640 1 -- 172.21.15.129:0/1662223268 wait complete. 2024-03-23T16:02:01.394 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-23T16:02:01.395 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-23T16:02:01.396 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T16:02:01.396 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:02:05.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-03-23T16:02:05.931+0000 b5e9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 41186) UID: 0 2024-03-23T16:02:05.937 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-03-23T16:02:05.933+0000 b5e9640 -1 mds.a *** got signal Terminated *** 2024-03-23T16:02:09.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:02:09.491 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:02:16.054 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-03-23T16:02:16.049+0000 a5e7640 -1 monclient: get_auth_request but no auth handler is set up 2024-03-23T16:02:39.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:02:39.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:02:46.972 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:06.816 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:46.986 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:16:06.785 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:46.987 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:16:06.786 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:46.989 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:16:06.788 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:47.021 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:06.865 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:47.037 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:16:06.836 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:47.101 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:06.945 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:47.182 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:16:06.981 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:02:47.184 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:16:06.983 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:09.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:03:09.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:03:38.717 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:58.560 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.722 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:58.567 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.765 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:16:58.609 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.776 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:16:58.575 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.785 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:16:58.584 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.812 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:16:58.611 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.817 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:16:58.616 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.819 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:16:58.618 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:38.859 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:16:58.658 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:03:39.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:03:39.807 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:04:03.876 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:23.720 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.879 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:23.723 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.909 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:17:23.753 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.941 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:17:23.740 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.942 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:17:23.742 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.967 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:17:23.766 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.975 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:17:23.774 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:03.978 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:17:23.778 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:04.010 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:17:23.809 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:04:09.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:04:09.985 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:04:40.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:04:40.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:05:10.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:05:10.296 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:05:30.345 INFO:tasks.ceph.mds.a.smithi129.stderr:==00:00:18:19.048 41201== 2024-03-23T16:05:30.346 INFO:tasks.ceph.mds.a.smithi129.stderr:==00:00:18:19.048 41201== Exit program on first error (--exit-on-first-error=yes) 2024-03-23T16:05:30.355 INFO:tasks.ceph.mds.a.smithi129.stderr:daemon-helper: command failed with exit status 42 2024-03-23T16:05:31.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-23T16:05:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-23T16:05:34.946 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a" 2024-03-23T16:05:34.947 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T16:05:34.947 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T16:05:34.947 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:05:34.960 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-03-23T16:05:34.950+0000 b5e9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c (PID: 41191) UID: 0 2024-03-23T16:05:34.961 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-03-23T16:05:34.952+0000 b5e9640 -1 mds.c *** got signal Terminated *** 2024-03-23T16:05:37.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-23T16:05:40.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:05:40.393 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:05:41.050 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T16:05:41.050 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-03-23T16:05:41.050 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:06:10.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:06:10.530 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:06:40.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:06:40.660 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:07:10.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:07:10.797 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:07:40.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:07:40.853 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:08:11.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:08:11.014 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:08:41.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:08:41.397 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:09:05.725 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-03-23T16:09:05.718+0000 139f9640 -1 MDSIOContextBase: failed with -108, restarting... 2024-03-23T16:09:11.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:09:11.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:09:38.051 INFO:tasks.ceph.mds.e.smithi129.stderr:2024-03-23T16:07:49.801+0000 b5e9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e (PID: 41195) UID: 0 2024-03-23T16:09:38.051 INFO:tasks.ceph.mds.e.smithi129.stderr:2024-03-23T16:07:49.802+0000 b5e9640 -1 mds.e *** got signal Terminated *** 2024-03-23T16:09:41.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:09:41.571 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:10:11.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:10:11.729 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:10:38.168 INFO:tasks.ceph.mds.f.smithi157.stderr: -1> 2024-03-23T16:09:05.718+0000 139f9640 -1 MDSIOContextBase: failed with -108, restarting... 2024-03-23T16:10:40.131 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-23T16:10:40.131 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-23T16:10:40.131 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:10:40.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-03-23T16:10:40.132+0000 7fc864bcc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 37459) UID: 0 2024-03-23T16:10:40.132 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-03-23T16:10:40.132+0000 7fc864bcc640 -1 osd.0 25 *** Got signal Terminated *** 2024-03-23T16:10:41.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:10:41.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:10:46.231 INFO:tasks.ceph.osd.0:Stopped 2024-03-23T16:10:46.232 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-23T16:10:46.232 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:10:46.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-03-23T16:10:46.231+0000 7f8bf2dba640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 37472) UID: 0 2024-03-23T16:10:46.232 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-03-23T16:10:46.231+0000 7f8bf2dba640 -1 osd.1 27 *** Got signal Terminated *** 2024-03-23T16:10:46.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-23T16:10:52.333 INFO:tasks.ceph.osd.1:Stopped 2024-03-23T16:10:52.333 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-23T16:10:52.333 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:10:52.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-03-23T16:10:52.333+0000 7fb895e73640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 37475) UID: 0 2024-03-23T16:10:52.334 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-03-23T16:10:52.333+0000 7fb895e73640 -1 osd.2 29 *** Got signal Terminated *** 2024-03-23T16:10:53.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-23T16:10:53.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-23T16:10:54.169 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:14.013 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.176 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:14.020 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.182 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:24:13.982 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.187 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:24:13.986 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.194 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:24:13.993 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.198 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:24:13.998 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.307 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:24:14.151 37202== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.349 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:24:14.146 37157== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:54.418 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:24:14.216 37156== Warning: unimplemented fcntl command: 1036 2024-03-23T16:10:58.435 INFO:tasks.ceph.osd.2:Stopped 2024-03-23T16:10:58.436 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-23T16:10:58.436 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:10:58.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T16:10:58.435+0000 7fbcecb3a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 37478) UID: 0 2024-03-23T16:10:58.437 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-03-23T16:10:58.435+0000 7fbcecb3a640 -1 osd.3 31 *** Got signal Terminated *** 2024-03-23T16:10:59.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-23T16:10:59.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-23T16:11:04.537 INFO:tasks.ceph.osd.3:Stopped 2024-03-23T16:11:04.537 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-23T16:11:04.537 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:04.538 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-23T16:11:04.536+0000 7fc77d213640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 37732) UID: 0 2024-03-23T16:11:04.538 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-23T16:11:04.536+0000 7fc77d213640 -1 osd.4 33 *** Got signal Terminated *** 2024-03-23T16:11:05.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-23T16:11:10.638 INFO:tasks.ceph.osd.4:Stopped 2024-03-23T16:11:10.639 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-23T16:11:10.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:10.639 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-23T16:11:10.638+0000 7fc2b224f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 37736) UID: 0 2024-03-23T16:11:10.640 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-23T16:11:10.638+0000 7fc2b224f640 -1 osd.5 35 *** Got signal Terminated *** 2024-03-23T16:11:11.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:11:11.919 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:11:16.740 INFO:tasks.ceph.osd.5:Stopped 2024-03-23T16:11:16.740 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-23T16:11:16.740 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:16.741 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-23T16:11:16.739+0000 7fdd621cf640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 37734) UID: 0 2024-03-23T16:11:16.741 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-23T16:11:16.739+0000 7fdd621cf640 -1 osd.6 37 *** Got signal Terminated *** 2024-03-23T16:11:22.842 INFO:tasks.ceph.osd.6:Stopped 2024-03-23T16:11:22.843 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-23T16:11:22.843 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:22.843 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-23T16:11:22.842+0000 7f681598c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 37737) UID: 0 2024-03-23T16:11:22.843 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-23T16:11:22.842+0000 7f681598c640 -1 osd.7 37 *** Got signal Terminated *** 2024-03-23T16:11:28.944 INFO:tasks.ceph.osd.7:Stopped 2024-03-23T16:11:28.944 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-23T16:11:28.945 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-23T16:11:28.945 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:28.969 INFO:tasks.ceph.mgr.y:Stopped 2024-03-23T16:11:28.970 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-23T16:11:28.970 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:28.970 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T16:11:28.969+0000 7fbb18e21640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 37154) UID: 0 2024-03-23T16:11:28.971 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-23T16:11:28.969+0000 7fbb18e21640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-23T16:11:28.995 INFO:tasks.ceph.mgr.x:Stopped 2024-03-23T16:11:28.996 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-23T16:11:28.996 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-23T16:11:28.996 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:11:29.024 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-03-23T16:11:29.013+0000 9f92640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 37180) UID: 0 2024-03-23T16:11:29.025 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-03-23T16:11:29.015+0000 9f92640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-23T16:11:41.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:11:41.978 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:12:12.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:12:12.123 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:12:42.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:12:42.307 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:13:12.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:13:12.436 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:13:42.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:13:42.742 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:14:12.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:14:12.824 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:14:42.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:14:42.955 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:11.507 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:31.351 37202== 2024-03-23T16:15:11.507 INFO:tasks.ceph.mon.a.smithi129.stderr:==00:00:28:31.351 37202== Exit program on first error (--exit-on-first-error=yes) 2024-03-23T16:15:11.519 INFO:tasks.ceph.mon.a.smithi129.stderr:daemon-helper: command failed with exit status 42 2024-03-23T16:15:12.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-23T16:15:13.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:13.090 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-23T16:15:14.754 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-23T16:15:14.756 INFO:tasks.ceph.mon.a:Stopped 2024-03-23T16:15:14.756 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-23T16:15:14.756 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:15:14.778 INFO:tasks.ceph.mon.b.smithi157.stderr:2024-03-23T16:15:14.761+0000 9f92640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 37149) UID: 0 2024-03-23T16:15:14.780 INFO:tasks.ceph.mon.b.smithi157.stderr:2024-03-23T16:15:14.763+0000 9f92640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-23T16:15:15.475 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:28:35.274 37156== 2024-03-23T16:15:15.475 INFO:tasks.ceph.mon.b.smithi157.stderr:==00:00:28:35.274 37156== Exit program on first error (--exit-on-first-error=yes) 2024-03-23T16:15:15.487 INFO:tasks.ceph.mon.b.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-23T16:15:17.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-23T16:15:17.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-23T16:15:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-23T16:15:20.858 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-23T16:15:20.859 INFO:tasks.ceph.mon.b:Stopped 2024-03-23T16:15:20.859 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-23T16:15:20.859 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T16:15:20.884 INFO:tasks.ceph.mon.c.smithi157.stderr:2024-03-23T16:15:20.864+0000 9f92640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 37152) UID: 0 2024-03-23T16:15:20.887 INFO:tasks.ceph.mon.c.smithi157.stderr:2024-03-23T16:15:20.866+0000 9f92640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-23T16:15:23.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-23T16:15:43.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:43.254 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:44.713 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:29:04.513 37157== 2024-03-23T16:15:44.714 INFO:tasks.ceph.mon.c.smithi157.stderr:==00:00:29:04.513 37157== Exit program on first error (--exit-on-first-error=yes) 2024-03-23T16:15:44.725 INFO:tasks.ceph.mon.c.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-23T16:15:45.266 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-23T16:15:45.266 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-23T16:15:45.267 INFO:tasks.ceph.mon.c:Stopped 2024-03-23T16:15:45.267 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-23T16:15:45.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-03-23T16:15:45.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-23T16:15:45.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T16:15:45.414 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-23T16:15:45.480 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T16:15:45.481 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-23T16:15:45.545 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T16:15:45.546 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-23T16:15:45.621 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T16:15:45.621 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-23T16:15:45.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T16:15:45.703 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-23T16:15:45.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T16:15:45.882 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-23T16:15:45.994 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T16:15:45.994 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-23T16:15:46.062 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T16:15:46.063 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-23T16:15:46.120 INFO:tasks.ceph:Archiving mon data... 2024-03-23T16:15:46.122 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/data/mon.a.tgz 2024-03-23T16:15:46.122 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-23T16:15:46.137 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.A3qhQfD3RW 2024-03-23T16:15:46.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.A3qhQfD3RW 2024-03-23T16:15:46.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.A3qhQfD3RW 2024-03-23T16:15:46.362 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.A3qhQfD3RW is 259KB 2024-03-23T16:15:46.457 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.A3qhQfD3RW 2024-03-23T16:15:46.472 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/data/mon.b.tgz 2024-03-23T16:15:46.472 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-23T16:15:46.486 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.KRpRrzWrLX 2024-03-23T16:15:46.487 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KRpRrzWrLX 2024-03-23T16:15:46.621 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.KRpRrzWrLX 2024-03-23T16:15:46.705 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.KRpRrzWrLX is 289KB 2024-03-23T16:15:46.771 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.KRpRrzWrLX 2024-03-23T16:15:46.785 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/data/mon.c.tgz 2024-03-23T16:15:46.785 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-23T16:15:46.840 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.G6H0BqqtXw 2024-03-23T16:15:46.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.G6H0BqqtXw 2024-03-23T16:15:46.965 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.G6H0BqqtXw 2024-03-23T16:15:47.048 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.G6H0BqqtXw is 299KB 2024-03-23T16:15:47.142 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.G6H0BqqtXw 2024-03-23T16:15:47.156 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-23T16:15:47.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T16:15:47.159 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T16:15:47.372 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-23T16:15:47.424 DEBUG:tasks.ceph:valgrind exception message: valgrind error: InvalidRead UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::leader_bootstrap(std::queue > >&&, std::chrono::duration >&) 2024-03-23T16:15:47.496 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-23T16:15:47.513 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-23T16:15:47.531 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-23T16:15:47.600 INFO:tasks.ceph:Archiving crash dumps... 2024-03-23T16:15:47.602 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/remote/smithi129/crash 2024-03-23T16:15:47.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T16:15:47.629 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/remote/smithi157/crash 2024-03-23T16:15:47.630 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T16:15:47.656 INFO:tasks.ceph:Shutting down logrotate 2024-03-23T16:15:47.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:47.675 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:47.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:47.766 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T16:15:47.805 INFO:tasks.ceph:Compressing logs... 2024-03-23T16:15:47.806 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T16:15:47.810 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T16:15:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48480.log 2024-03-23T16:15:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57883.log 2024-03-23T16:15:47.835 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57043.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37390.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37390.log.gz 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37430.log: /var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-23T16:15:47.903 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/mds.f.log: /var/log/ceph/valgrind/mds.b.log: 55.3% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr: 55.2% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37470.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/valgrind/mds.d.log: 69.6%gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr: 55.2% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37470.log: /var/log/ceph/ceph.audit.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37470.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37510.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37550.log 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37510.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37510.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr:gzip 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-23T16:15:47.904 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-23T16:15:47.905 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37550.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-23T16:15:47.905 INFO:teuthology.orchestra.run.smithi157.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37550.log.gz 2024-03-23T16:15:47.905 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-23T16:15:47.905 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.5.log: 91.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-23T16:15:47.905 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-23T16:15:47.921 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-23T16:15:47.921 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mgr.x.log: gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping 2024-03-23T16:15:47.921 INFO:teuthology.orchestra.run.smithi157.stderr: 89.2% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-23T16:15:47.921 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37595.log 2024-03-23T16:15:47.921 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.log: 93.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-23T16:15:47.922 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-23T16:15:47.922 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37595.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37595.log.gz 2024-03-23T16:15:47.922 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37350.log 2024-03-23T16:15:47.930 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37641.log 2024-03-23T16:15:47.931 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37350.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37350.log.gz 2024-03-23T16:15:47.931 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-23T16:15:47.931 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37641.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37641.log.gz 2024-03-23T16:15:47.931 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-23T16:15:47.932 INFO:teuthology.orchestra.run.smithi157.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-23T16:15:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43441.log 2024-03-23T16:15:48.009 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48480.log.gz 2024-03-23T16:15:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57505.log 2024-03-23T16:15:48.009 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57883.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54560.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57043.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4%/var/log/ceph/ceph-client.admin.57505.log: -- replaced with /var/log/ceph/ceph-client.admin.43441.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50610.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57505.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56959.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50610.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54560.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50610.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45827.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.a.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56959.log.gz 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43399.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42528.log 2024-03-23T16:15:48.010 INFO:teuthology.orchestra.run.smithi129.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45827.log.gz 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48210.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43399.log.gz 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54896.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42528.log: /var/log/ceph/ceph-client.admin.48210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46349.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42528.log.gz 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48210.log.gz 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54896.log.gz 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46139.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46349.log.gz 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56026.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41461.log: /var/log/ceph/ceph-client.admin.46139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53768.log 2024-03-23T16:15:48.011 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56026.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46139.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53810.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56026.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56875.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53768.log: /var/log/ceph/ceph-client.admin.53810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53768.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56875.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53810.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54980.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56875.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47958.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54980.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49818.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45137.log 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-03-23T16:15:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49818.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47958.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47706.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49818.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49296.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45137.log: /var/log/ceph/ceph-client.admin.47706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45137.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49296.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47706.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49254.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49296.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46830.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54290.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52118.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46830.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49254.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54290.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46830.log.gz 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42531.log 2024-03-23T16:15:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54290.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54770.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52118.log: /var/log/ceph/ceph-client.admin.42531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43315.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52118.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54770.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42531.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46746.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54770.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51638.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43315.log: /var/log/ceph/ceph-client.admin.46746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46662.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43315.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51638.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46746.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54686.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51638.log.gz 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52868.log 2024-03-23T16:15:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46662.log: /var/log/ceph/ceph-client.admin.54686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46662.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57799.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54686.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42524.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52868.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54602.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44885.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57799.log: /var/log/ceph/ceph-client.admin.54602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55900.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57799.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44885.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54602.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51259.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44885.log.gz 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45953.log 2024-03-23T16:15:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55900.log: /var/log/ceph/ceph-client.admin.51259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54248.log 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55900.log.gz 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51259.log.gz 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48960.log 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45953.log.gz 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53726.log/var/log/ceph/ceph-client.admin.54248.log: 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50989.log 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54248.log.gz 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53726.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48960.log.gz 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53726.log.gz 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51908.log 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50989.log: /var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45449.log 2024-03-23T16:15:48.016 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50989.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51908.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54080.log 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51908.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52826.log 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45449.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46223.log 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54080.log: /var/log/ceph/ceph-client.admin.52826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57421.log 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9%/var/log/ceph/ceph-client.admin.46223.log: -- replaced with /var/log/ceph/ceph-client.admin.54080.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52826.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51596.log 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46223.log.gz 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45869.log 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45785.log/var/log/ceph/ceph-client.admin.51596.log: 2024-03-23T16:15:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57421.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45869.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51596.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45869.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45785.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45785.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/valgrind/mds.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: 57.9% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.e.log 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/valgrind/mds.a.log: /var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- 64.2% /var/log/ceph/ceph-client.admin.53162.log 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/valgrind/mds.a.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-23T16:15:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/valgrind/mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56791.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 55.2% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42116.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53162.log: /var/log/ceph/ceph-client.admin.56791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52598.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53162.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7%/var/log/ceph/ceph-client.admin.42116.log: -- replaced with /var/log/ceph/ceph-client.admin.56791.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44164.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42116.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53288.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52598.log: /var/log/ceph/ceph-client.admin.44164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53954.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52598.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53288.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44164.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53288.log.gz 2024-03-23T16:15:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53684.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53954.log.gz 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54728.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52076.log: /var/log/ceph/ceph-client.admin.53684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51512.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53684.log.gz 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50508.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54728.log.gz 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48312.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51512.log: /var/log/ceph/ceph-client.admin.50508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52304.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51512.log.gz 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50508.log.gz 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45617.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42911.log 2024-03-23T16:15:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52304.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48312.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45617.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52304.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56647.log 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45617.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42911.log: /var/log/ceph/ceph-client.admin.56647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55984.log 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42911.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50947.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56647.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55816.log 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42218.log 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43609.log 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55816.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55984.log.gz 2024-03-23T16:15:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42218.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.55816.log.gz 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44783.log 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42218.log.gz 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44927.log 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43609.log: /var/log/ceph/ceph-client.admin.44783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51824.log 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43609.log.gz 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44927.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44783.log.gz 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45659.log 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44927.log.gz 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48876.log 2024-03-23T16:15:48.022 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51824.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46013.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45659.log: /var/log/ceph/ceph-client.admin.48876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54938.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45659.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46013.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48876.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51133.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41939.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46013.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54938.log: /var/log/ceph/ceph-client.admin.51133.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54938.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55292.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51133.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41939.log: /var/log/ceph/ceph-client.admin.55292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46055.log 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41939.log.gz 2024-03-23T16:15:48.023 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55292.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53036.log 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44122.log 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46055.log: /var/log/ceph/ceph-client.admin.53036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42306.log 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44122.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46055.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53036.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54458.log 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44122.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50256.log 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42306.log: /var/log/ceph/ceph-client.admin.54458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50130.log 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42306.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54458.log.gz 2024-03-23T16:15:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49212.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50466.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50130.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50256.log.gz 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49566.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50130.log.gz 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49212.log.gz 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43735.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50466.log: /var/log/ceph/ceph-client.admin.49566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47394.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50466.log.gz 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43735.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49566.log.gz 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42915.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43735.log.gz 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48648.log 2024-03-23T16:15:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47394.log: /var/log/ceph/ceph-client.admin.42915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49776.log 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47394.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42915.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57547.log 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56152.log 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49776.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48648.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51175.log 88.1% 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49776.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57547.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44531.log 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53558.log 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56152.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44531.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51175.log.gz 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44405.log 2024-03-23T16:15:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44531.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55858.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53558.log: /var/log/ceph/ceph-client.admin.44405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48168.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53558.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44405.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46704.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55774.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48168.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55858.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46704.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48168.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56254.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46704.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45389.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55774.log: /var/log/ceph/ceph-client.admin.56254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54038.log 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55774.log.gz 2024-03-23T16:15:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45389.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56254.log.gz 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50568.log 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45389.log.gz 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56689.log 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54038.log.gz 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57841.log 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50568.log: /var/log/ceph/ceph-client.admin.56689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56380.log 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50568.log.gz 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57841.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56689.log.gz 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58028.log 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56380.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57841.log.gz 2024-03-23T16:15:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48042.log: gzip -5 --verbose 87.9% -- -- replaced with /var/log/ceph/ceph-client.admin.56380.log.gz /var/log/ceph/ceph-client.admin.44447.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54416.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58028.log: /var/log/ceph/ceph-client.admin.44447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51428.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58028.log.gz 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54416.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44447.log.gz 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56068.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54416.log.gz/var/log/ceph/ceph-client.admin.51428.log: 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45575.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51428.log.gz 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47082.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56068.log: /var/log/ceph/ceph-client.admin.45575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48270.log 2024-03-23T16:15:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56068.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47082.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45575.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55588.log 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47082.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45011.log 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48270.log: /var/log/ceph/ceph-client.admin.55588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47832.log 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48270.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45011.log: gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55588.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph.audit.log 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47832.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45011.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44080.log 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47832.log.gz 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46577.log 2024-03-23T16:15:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.44080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52682.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44080.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51031.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52682.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46577.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52682.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51031.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48396.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48396.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52430.log 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48396.log.gz 2024-03-23T16:15:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51302.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41287.log: /var/log/ceph/ceph-client.admin.52430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49044.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52430.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46307.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52994.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51302.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49044.log: /var/log/ceph/ceph-client.admin.46307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47520.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49044.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46307.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52994.log: -- /var/log/ceph/ceph-client.admin.45305.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55124.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52994.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45305.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47520.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44969.log 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4%/var/log/ceph/ceph-client.admin.55124.log: -- replaced with /var/log/ceph/ceph-client.admin.45305.log.gz 2024-03-23T16:15:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53474.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6%/var/log/ceph/ceph-client.admin.44969.log: -- replaced with /var/log/ceph/ceph-client.admin.55124.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53330.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53474.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44969.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48918.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53474.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53330.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47352.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48918.log: /var/log/ceph/ceph-client.admin.41245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55082.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48918.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47352.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55082.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47352.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55166.log 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55082.log.gz 2024-03-23T16:15:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53894.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54374.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41981.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53894.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55166.log.gz 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54374.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53894.log.gz 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47310.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50214.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41981.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54374.log.gz 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47310.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.41981.log.gz 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52556.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42263.log 2024-03-23T16:15:48.034 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47310.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50214.log: /var/log/ceph/ceph-client.admin.52556.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50214.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44363.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52556.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52472.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42263.log: /var/log/ceph/ceph-client.admin.44363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53600.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42263.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52472.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44363.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53600.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52472.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46475.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53600.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57211.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46788.log 2024-03-23T16:15:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46475.log: /var/log/ceph/ceph-client.admin.57211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46475.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57211.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50088.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49944.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43777.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46788.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50088.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37483.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50088.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46391.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49944.log: /var/log/ceph/ceph-client.admin.37483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57463.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49944.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37483.log.gz 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50821.log 2024-03-23T16:15:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56731.log 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57463.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53246.log 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57463.log.gz 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50821.log.gz 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43083.log 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42761.log 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56731.log.gz 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43083.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53246.log.gz 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47142.log 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42761.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43083.log.gz 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-23T16:15:48.037 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42761.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48816.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47142.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45263.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.48816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57967.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45263.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48816.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57715.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57967.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45263.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51386.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57715.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57967.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48690.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3%/var/log/ceph/ceph-client.admin.51386.log: -- replaced with /var/log/ceph/ceph-client.admin.57715.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47436.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51386.log.gz 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49860.log 2024-03-23T16:15:48.038 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48690.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55546.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49860.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47436.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42349.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49860.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44699.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52034.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42349.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55546.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42349.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52034.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44699.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56212.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52034.log.gz 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50905.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57169.log 2024-03-23T16:15:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56212.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56422.log 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56212.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.57169.log: -- /var/log/ceph/ceph-client.admin.43651.log 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50905.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56422.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57169.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46998.log 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43651.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56422.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43651.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46872.log 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4%/var/log/ceph/ceph-client.admin.46998.log: -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-03-23T16:15:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45491.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46998.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51950.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47226.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46872.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51950.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46872.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51950.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47226.log: /var/log/ceph/ceph-client.admin.41085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50737.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47226.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42517.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50779.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49002.log 2024-03-23T16:15:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53642.log 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50779.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50737.log.gz 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57379.log 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50779.log.gz 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49002.log.gz 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54332.log 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53642.log: /var/log/ceph/ceph-client.admin.57379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49170.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53642.log.gz 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57379.log.gz 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51866.log 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54332.log.gz 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55461.log 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53432.log 2024-03-23T16:15:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49170.log.gz 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51866.log: /var/log/ceph/ceph-client.admin.55461.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43903.log 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56917.log 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55461.log.gz 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43903.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53432.log.gz 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54812.log 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54518.log 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56917.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43903.log.gz 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54812.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.56917.log.gz -5 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43525.log 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54812.log.gz 2024-03-23T16:15:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42514.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54518.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42917.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43525.log: /var/log/ceph/ceph-client.admin.51698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57127.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5%/var/log/ceph/ceph-client.admin.42917.log: -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47916.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57589.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42917.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47916.log: gzip -5 87.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.57127.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-osd.3.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr: --/var/log/ceph/ceph-client.admin.57589.log: /var/log/ceph/ceph-client.admin.44249.log 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.3.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57589.log.gz 2024-03-23T16:15:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55942.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44249.log: /var/log/ceph/ceph-client.admin.55942.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.52262.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48000.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55942.log.gz 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45179.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50424.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45179.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52262.log.gz 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46433.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50424.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45179.log.gz 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43861.log 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50424.log.gz 2024-03-23T16:15:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43181.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46433.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43861.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46097.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43181.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43861.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 88.0% --verbose -- /var/log/ceph/ceph-client.admin.44741.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43181.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47604.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44741.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55376.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47604.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55648.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44741.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47604.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57295.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56110.log 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55376.log.gz 2024-03-23T16:15:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46619.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55648.log: /var/log/ceph/ceph-client.admin.56110.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57295.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55504.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57673.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55648.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56110.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55334.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46619.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55504.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.57673.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55334.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57673.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55334.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.53852.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53390.log 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41758.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2024-03-23T16:15:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48354.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53390.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44207.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.48354.log: 88.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.57631.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44207.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48354.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52160.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44207.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48774.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57631.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.52160.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50046.log 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48774.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52160.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48774.log.gz 2024-03-23T16:15:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.53078.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50172.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50046.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53078.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53078.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45347.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56833.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50172.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49524.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56833.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45347.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50298.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56833.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56548.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50298.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42023.log 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49524.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56548.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50298.log.gz 2024-03-23T16:15:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51992.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56548.log.gz 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49338.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57085.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51992.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42023.log.gz 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43265.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51992.log.gz 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49338.log.gz 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52724.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57085.log: /var/log/ceph/ceph-client.admin.43265.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57085.log.gz 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50004.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43265.log.gz 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45095.log 2024-03-23T16:15:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52724.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52640.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50004.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45095.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52640.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45095.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55418.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mds.e.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52640.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52952.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50695.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52952.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55418.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54854.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52952.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50695.log: /var/log/ceph/ceph-client.admin.54854.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50695.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54854.log.gz 2024-03-23T16:15:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47562.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44657.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43143.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47562.log.gz 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44657.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44657.log.gz 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47748.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48522.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43143.log.gz/var/log/ceph/ceph-client.admin.47748.log: 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51740.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz/var/log/ceph/ceph-client.admin.48522.log: 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45743.log 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48522.log.gz 2024-03-23T16:15:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48732.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51740.log: /var/log/ceph/ceph-client.admin.45743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57001.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51740.log.gz 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48732.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45743.log.gz 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48732.log.gz 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42526.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57001.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.58068.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57001.log.gz 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49380.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42526.log.gz 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55690.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49380.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58068.log.gz 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42912.log 2024-03-23T16:15:48.053 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49380.log.gz 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55690.log: /var/log/ceph/ceph-client.admin.42912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47478.log 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41200.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55690.log.gz 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42912.log.gz 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47268.log 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47478.log: /var/log/ceph/ceph-client.admin.47268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51217.log 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47478.log.gz 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41590.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47268.log.gz 2024-03-23T16:15:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43693.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51217.log.gz 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57337.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56506.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43693.log.gz 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51344.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56506.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57337.log.gz 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48438.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56506.log.gz 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50653.log 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48438.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51344.log.gz 2024-03-23T16:15:48.055 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48438.log.gz 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41802.log 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42158.log 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41802.log: /var/log/ceph/ceph-client.admin.50653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49608.log 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50653.log.gz 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42158.log: -- replaced with /var/log/ceph/ceph-client.admin.41802.log.gz 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.53516.log 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42158.log.gz 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49608.log.gz/var/log/ceph/ceph-client.admin.53516.log: 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43357.log 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53516.log.gz 2024-03-23T16:15:48.056 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42065.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43357.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43357.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53204.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50340.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42065.log: /var/log/ceph/ceph-client.admin.53204.log: 88.3% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53204.log.gz 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42065.log.gz 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45053.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56296.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50340.log.gz 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55208.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56296.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45053.log.gz 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52346.log 2024-03-23T16:15:48.057 INFO:teuthology.orchestra.run.smithi129.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56296.log.gz 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49128.log 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52346.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55208.log.gz 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52346.log.gz 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48084.log 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5%/var/log/ceph/ceph-client.admin.49692.log: -- replaced with /var/log/ceph/ceph-client.admin.49128.log.gz 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49734.log 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50382.log 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48084.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53996.log 2024-03-23T16:15:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49734.log: /var/log/ceph/ceph-client.admin.50382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46914.log 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53996.log: 88.0% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49734.log.gz 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50382.log.gz 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53996.log.gz 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52514.log 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46914.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47646.log 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46914.log.gz 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.52514.log.gz 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47184.log 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47646.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-23T16:15:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47184.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47646.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47184.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44489.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54122.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44489.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr: 88.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54206.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54122.log: /var/log/ceph/ceph-client.admin.54206.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54122.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54206.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42516.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mds.c.log: 93.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45911.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46956.log 2024-03-23T16:15:48.060 INFO:teuthology.orchestra.run.smithi129.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42516.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45911.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45911.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52910.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51073.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46956.log: /var/log/ceph/ceph-client.admin.52910.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46956.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52910.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44825.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48564.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44825.log: /var/log/ceph/ceph-client.admin.51073.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44825.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51073.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42722.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52202.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42722.log: /var/log/ceph/ceph-client.admin.48564.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42722.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48564.log.gz 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51782.log 2024-03-23T16:15:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52202.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52202.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54644.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45221.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54644.log: /var/log/ceph/ceph-client.admin.51782.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51782.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54644.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48606.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45221.log: /var/log/ceph/ceph-client.admin.48606.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45221.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48606.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57253.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55250.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45701.log 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57253.log: /var/log/ceph/ceph-client.admin.49650.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57253.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2024-03-23T16:15:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49902.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45701.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45701.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55732.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56338.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49902.log: /var/log/ceph/ceph-client.admin.55732.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49902.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55732.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49086.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56338.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56338.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43567.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41548.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49086.log: /var/log/ceph/ceph-client.admin.43567.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49086.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43567.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41548.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41548.log.gz 2024-03-23T16:15:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55022.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37219.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47790.log: /var/log/ceph/ceph-client.admin.55022.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55022.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41716.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37219.log: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.37219.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31192.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56464.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41716.log: /var/log/ceph/ceph.tmp-client.admin.31192.log: 0.0% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41716.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.31192.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41674.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56464.log: /var/log/ceph/ceph-client.admin.41851.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56464.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50863.log 2024-03-23T16:15:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41674.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41674.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54164.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50863.log: /var/log/ceph/ceph-client.admin.54164.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50863.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54164.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37594.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46517.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57925.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47040.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37594.log: /var/log/ceph/ceph-client.admin.57925.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37594.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57925.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47040.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47040.log.gz 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-03-23T16:15:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42735.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41506.log: /var/log/ceph/ceph-client.admin.43945.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49482.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42735.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42735.log.gz 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52388.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49482.log: /var/log/ceph/ceph-client.admin.52388.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49482.log.gz 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52388.log.gz 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52766.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41361.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43223.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-23T16:15:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52766.log: /var/log/ceph/ceph-client.admin.43223.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52766.log.gz 2024-03-23T16:15:48.067 INFO:teuthology.orchestra.run.smithi129.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43223.log.gzgzip 2024-03-23T16:15:48.067 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45533.log 2024-03-23T16:15:48.067 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53120.log 2024-03-23T16:15:48.067 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45533.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45533.log.gz 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51554.log 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53120.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53120.log.gz 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51554.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51554.log.gz 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42913.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48126.log 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43483.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44573.log 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48126.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48126.log.gz 2024-03-23T16:15:48.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57757.log 2024-03-23T16:15:48.074 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44573.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44573.log.gz 2024-03-23T16:15:48.074 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57757.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57757.log.gz 2024-03-23T16:15:48.074 INFO:teuthology.orchestra.run.smithi129.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-23T16:15:48.322 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.c.log: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-23T16:15:48.372 INFO:teuthology.orchestra.run.smithi157.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-23T16:15:48.675 INFO:teuthology.orchestra.run.smithi157.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-23T16:15:48.734 INFO:teuthology.orchestra.run.smithi129.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-23T16:15:48.803 INFO:teuthology.orchestra.run.smithi129.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-23T16:15:48.878 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-23T16:15:49.026 INFO:teuthology.orchestra.run.smithi129.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-23T16:15:49.039 INFO:teuthology.orchestra.run.smithi129.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-23T16:15:49.041 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-23T16:15:49.041 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m1.219s 2024-03-23T16:15:49.041 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m5.047s 2024-03-23T16:15:49.041 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.401s 2024-03-23T16:15:49.080 INFO:teuthology.orchestra.run.smithi157.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-23T16:15:49.212 INFO:teuthology.orchestra.run.smithi157.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-23T16:15:49.287 INFO:teuthology.orchestra.run.smithi157.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-23T16:15:49.315 INFO:teuthology.orchestra.run.smithi157.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-23T16:15:49.317 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-23T16:15:49.317 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m1.455s 2024-03-23T16:15:49.317 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m5.973s 2024-03-23T16:15:49.317 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m0.223s 2024-03-23T16:15:49.318 INFO:tasks.ceph:Archiving logs... 2024-03-23T16:15:49.318 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/remote/smithi129/log 2024-03-23T16:15:49.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T16:15:49.486 INFO:tasks.ceph.mds.f.smithi157.stderr:==00:00:28:38.172 41336== 2024-03-23T16:15:49.486 INFO:tasks.ceph.mds.f.smithi157.stderr:==00:00:28:38.172 41336== Exit program on first error (--exit-on-first-error=yes) 2024-03-23T16:15:49.584 INFO:tasks.ceph.mds.f.smithi157.stderr:daemon-helper: command failed with exit status 42 2024-03-23T16:15:50.312 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/remote/smithi157/log 2024-03-23T16:15:50.313 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T16:15:50.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-03-23T16:15:50.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-23T16:15:51.051 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: InvalidRead UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::leader_bootstrap(std::queue > >&&, std::chrono::duration >&) 2024-03-23T16:15:54.733 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=24767e64637f4d218dfc2157d736f181 Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: InvalidRead UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::leader_bootstrap(std::queue > >&&, std::chrono::duration >&) 2024-03-23T16:15:54.738 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-23T16:15:54.747 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T16:15:54.756 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T16:15:54.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T16:15:54.771 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T16:15:54.847 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T16:15:54.856 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T16:15:54.856 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T16:15:54.859 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T16:15:54.870 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-23T16:15:54.873 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-23T16:15:54.874 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T16:15:54.874 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-23T16:15:54.874 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +2831us[+2831us] +/- 91ms 2024-03-23T16:15:54.874 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1027us[-1034us] +/- 56ms 2024-03-23T16:15:54.874 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -935us[ -942us] +/- 81ms 2024-03-23T16:15:54.874 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T16:15:54.876 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T16:15:54.877 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-23T16:15:54.877 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +2825us[+2825us] +/- 91ms 2024-03-23T16:15:54.877 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -1022us[-1022us] +/- 56ms 2024-03-23T16:15:54.877 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 38 -916us[ -917us] +/- 81ms 2024-03-23T16:15:54.877 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T16:15:54.878 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T16:15:54.889 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T16:15:54.889 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T16:15:54.897 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T16:15:54.988 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T16:15:55.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T16:15:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T16:15:55.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T16:15:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T16:15:55.114 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T16:15:55.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T16:15:55.135 INFO:teuthology.task.internal:Duration was 2560.166898 seconds 2024-03-23T16:15:55.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T16:15:55.144 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T16:15:55.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T16:15:55.147 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T16:15:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T16:15:55.197 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T16:15:55.607 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T16:15:55.607 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-23T16:15:55.607 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T16:15:55.627 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-03-23T16:15:55.627 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T16:15:55.668 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T16:15:55.668 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T16:15:55.671 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T16:15:55.738 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T16:15:55.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T16:15:55.741 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T16:15:55.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T16:15:55.920 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T16:15:55.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T16:15:55.948 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T16:15:55.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T16:15:55.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T16:15:55.992 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T16:15:56.015 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-23T16:15:56.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~5s 2024-03-23T16:15:56.042 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-03-23T16:15:56.067 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T16:15:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T16:15:56.090 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T16:15:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T16:15:56.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T16:15:56.131 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T16:15:56.132 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/remote/smithi129 2024-03-23T16:15:56.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T16:15:56.169 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460/remote/smithi157 2024-03-23T16:15:56.169 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T16:15:56.205 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T16:15:56.205 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T16:15:56.208 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T16:15:56.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T16:15:56.265 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T16:15:56.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T16:15:56.274 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T16:15:56.274 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T16:15:56.276 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T16:15:56.288 INFO:teuthology.orchestra.run.smithi129.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 16:15 /home/ubuntu/cephtest 2024-03-23T16:15:56.311 INFO:teuthology.orchestra.run.smithi157.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 16:15 /home/ubuntu/cephtest 2024-03-23T16:15:56.312 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T16:15:56.325 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T16:15:56.354 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T16:15:56.379 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-23T16:15:56.380 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T16:15:56.404 DEBUG:teuthology.nuke:shortname: smithi157 2024-03-23T16:15:56.405 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T16:15:56.409 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 15:23:59.582309', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARsO9mkf9Sa31l8h16xrU7JaFkB1i1hbbhND5Jh393VvdrfJQgScMrL5q2WDTpBNPcCZECoHnbd2bnmqxUjiXQ='} 2024-03-23T16:15:56.430 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 15:23:59.580885', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+9px7MnSxpoSkDwUcXr7nRW8kSItohx68roYhA2tWLsD7fcV7x+aE719oUsVNHglGXaCzm4ON+32TRNsGUUKg='} 2024-03-23T16:15:56.462 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-23T16:15:56.463 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T16:15:56.488 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-23T16:15:56.489 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:15:56.509 INFO:teuthology.orchestra.console.smithi157:Power off 2024-03-23T16:15:56.509 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T16:15:56.522 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-23T16:15:56.537 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2024-03-23T16:15:56.538 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:15:56.564 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-23T16:16:00.523 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:16:00.565 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:16:00.634 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-23T16:16:00.676 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-23T16:16:01.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-03-23T16:16:01.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-23T16:16:01.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-23T16:16:01.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~11s 2024-03-23T16:16:04.638 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:16:04.677 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:16:04.749 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-23T16:16:04.788 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2024-03-23T16:16:04.888 INFO:teuthology.orchestra.console.smithi157:Power off completed 2024-03-23T16:16:06.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~5s 2024-03-23T16:16:06.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~5s 2024-03-23T16:16:06.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~5s 2024-03-23T16:16:06.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~16s 2024-03-23T16:16:08.753 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T16:16:08.863 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-23T16:16:08.964 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-23T16:16:09.069 INFO:teuthology.run:Summary data: description: fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} duration: 2560.166897535324 failure_reason: 'valgrind error: InvalidRead UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::leader_bootstrap(std::queue > >&&, std::chrono::duration >&)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=24767e64637f4d218dfc2157d736f181 status: fail success: false 2024-03-23T16:16:09.070 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T16:16:09.155 INFO:teuthology.run:FAIL